Use a representative: Which approach satisfies that requirement?
Use a representative test set kept untouched throughout development.
The question
A research-administration model is being evaluated on applications from the same period used for training. Approval requires an unbiased estimate of performance on unseen applications, using a test population representative of the intended applications and untouched by training, tuning, feature selection, or threshold choice. Which approach satisfies that requirement?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Compare predictions with administrator decisions from the same periodContemporaneous administrator decisions may be reused, selective, or biased labels and do not create an untouched representative test set.
- Use a representative, untouched test set ✓A representative test set kept untouched throughout development provides an estimate on unseen applications and avoids development contamination.
- Tune the model further on current applications before scoring themUsing the applications for further tuning means they are no longer untouched and can produce an optimistically biased estimate.
- Report repeated cross-validation on all current applicationsCross-validation can be useful, but repeated reuse of these applications does not satisfy the stated requirement for an untouched representative test population.
The trap
Check both dimensions: test records must be untouched and must represent the population to which performance will be generalized. How to remember it
Use a representative test set kept untouched throughout development.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Exclude the proxy and validate subgroup performance: Which approach meets the condition? →
- Deduplicate records before creating train and test: Which approach should replace the current evaluation? →
- Measure each subgroup's false-positive rate against 5%: Which evaluation approach best satisfies that →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.
