Use subgroup acceptance criteria and investigate: What evidence should determine approval?
Use subgroup-specific acceptance criteria and investigate accessibility-related errors.
The question
A travel-support classifier routes requests. It achieves 96% overall accuracy: 99% for routine itinerary changes and 82% for accessibility-related requests, which are important to the deployment population. What evidence should determine approval?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Report the overall accuracy with confidence intervalsConfidence intervals quantify uncertainty around the average but do not resolve the accessibility subgroup’s substantially lower performance.
- Compare aggregate accuracy across additional travel seasonsSeasonal aggregate results may reveal temporal variation but can continue to mask accessibility-specific errors.
- Use subgroup acceptance criteria and investigate accessibility errors ✓Approval should address the materially lower-performing accessibility subgroup using explicit criteria and error analysis, rather than relying on the overall average.
- Expand routine-request testing to stabilize the overall estimateMore routine cases may improve the aggregate estimate while leaving the important accessibility subgroup inadequately evaluated.
The trap
When an important subgroup performs materially worse, evaluate it separately instead of relying on the overall average. How to remember it
Use subgroup-specific acceptance criteria and investigate accessibility-related errors.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Create a deduplicated held-out clip set: Which evidence should reviewers request? →
- Control false negatives: Which evidence should receive priority before approval? →
- Conduct security testing alongside targeted robustness: Which plan is most defensible? →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.
