Split by policyholder before testing: What correction is decisive?
Group related records before splitting so policyholders cannot appear across development and evaluation, then examine residual linkage risks.
The question
An insurer’s claim model uses multiple records per policyholder. Engineers randomly split rows, so related claims from the same people appear in both training and testing. The test set must remain useful and policyholder privacy must be respected. What correction is decisive?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Remove identifiers after splitting so related records cannot be recognized.Removing identifiers after splitting does not undo leakage, because related records already influenced both model development and evaluation.
- Split by policyholder before testing, then inspect duplicate and linkage risks. ✓Grouping records by policyholder prevents direct cross-set duplication while linkage inspection addresses remaining leakage and privacy concerns.
- Keep the split and report confidence intervals around its accuracy.Uncertainty intervals describe sampling variation but do not correct dependency or information leakage between training and testing records.
- Increase the test-set percentage while retaining row-level random splitting.A larger contaminated test set still permits related policyholder records to leak information across evaluation boundaries.
The trap
Choose the unit of splitting that matches the dependence structure, not automatically the individual row. How to remember it
Group related records before splitting so policyholders cannot appear across development and evaluation, then examine residual linkage risks.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Use a representative test set untouched by training: What evidence is decisive before deployment? →
- Report subgroup-specific false-negative rates alongside: Which analysis is decisive? →
- Reduce false negatives first: Which evaluation priority follows? →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.
