Measure each subgroup's false-positive rate against 5%: Which evaluation approach best satisfies that

Measure false-positive rates separately for every subgroup against the 5% limit.

The question

Preparing for AIGP? Take the free 5-min readiness quiz →

1. Compare aggregate precision and recall across several yearly test sets.

   Yearly aggregate precision and recall assess stability but can still hide false-positive disparities among student subgroups.
2. Set one threshold from the population-wide confusion matrix and document its validation.

   A population-wide threshold may be operationally simple, but its aggregate confusion matrix cannot demonstrate that every subgroup is below 5%.
3. Report held-out overall accuracy.

   Overall accuracy can meet its target while concealing subgroup false-positive disparities, so it does not test the stated constraint.
4. Measure each subgroup's false-positive rate against 5%. ✓

   This directly evaluates the metric and threshold required for every documented subgroup, rather than relying on aggregate performance.

How to remember it

Measure false-positive rates separately for every subgroup against the 5% limit.

More Understanding How to Govern AI Development questions

• Deduplicate records before creating train and test: Which approach should replace the current evaluation? →
• Use a cost-sensitive threshold: Which approach best addresses both the asymmetric error costs and limited →
• Validate captions on representative data and test leakage: Which validation plan best addresses these →
• All 426 Understanding How to Govern AI Development questions →