Validation and test data held out from training: To produce this evidence, on which data should the decisive

Generalization is demonstrated by evaluating on validation and test data held out from training.

The question

Preparing for AIGP? Take the free 5-min readiness quiz →

1. The full training dataset reused for scoring, since evaluating on all examples yields the largest sample.

   Plausible but wrong: scoring on training data rewards memorization and inflates results, so it cannot demonstrate generalization.
2. Validation and test data held out from training, so measured performance reflects how the model behaves on inputs it has not seen. ✓

   Correct: evaluating on held-out validation and test data is how a team gauges generalization to unseen inputs.
3. A synthetic dataset generated to mirror the training data's statistics, so evaluation avoids any dependence on real held-out records.

   Plausible but wrong: synthetic data echoing training statistics still fails to show behavior on genuinely unseen real inputs.
4. Only the small subset of edge cases the team found hardest, since strong results there imply the model handles everything else well.

   Plausible but wrong: a hand-picked edge-case slice is not a representative held-out set and cannot establish overall generalization.

How to remember it

Generalization is demonstrated by evaluating on validation and test data held out from training.

More Understanding How to Govern AI Development questions

• Establishing provenance by documenting the data's origin: Applying data governance, which practice most →
• Document the proxy-bias and overfitting findings: Which action best fits a training-and-testing →
• A record of datasets: Which artifact most directly satisfies this need? →
• All 426 Understanding How to Govern AI Development questions →