Use a representative test set untouched by training | AIGP
7-day money-back guarantee — full refund within 7 days of purchase if you've completed under 20% of the questions. See pricing →
Certifications Tools Flashcards Career Paths Exam Guides Blog Pricing For Teams About

Language

✓ EnglishDeutschEspañolFrançaisPortuguês
Check readiness — free →

Use a representative test set untouched by training: What evidence is decisive before deployment?

AIGP Understanding How to Govern AI Development Hard

Require representative test data untouched by training or tuning before relying on performance results.

The question

An employment-screening vendor reports 95% accuracy after tuning thresholds on a dataset that includes records used for training. The vendor cannot identify which examples influenced tuning. What evidence is decisive before deployment?

Preparing for AIGP? Take the free 5-min readiness quiz →

  1. Base deployment on subgroup metrics from the contaminated records.
    Subgroup metrics are important, but they remain unreliable when the records influenced training or threshold selection.
  2. Accept the accuracy target as sufficient evidence.
    A favorable metric from training-influenced records may overstate performance on unseen applicants.
  3. Use a representative test set untouched by training and tuning. ✓
    An independent representative test set can provide meaningful evidence of generalization to intended applicants.
  4. Collect more examples from the same mixed dataset.
    More data do not restore independence when training and evaluation influence cannot be separated.
The trap
Treat evaluation as compromised when development decisions may have used the same records.

How to remember it

Require representative test data untouched by training or tuning before relying on performance results.

How many of these would you get right?

One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.

Test your AIGP readiness — free

More Understanding How to Govern AI Development questions

Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development · Every answer, right and wrong, comes with its own explanation.