Adjudicate a stratified disagreement sample using: Which evidence most directly resolves that uncertainty?
A stratified adjudication sample with an operational rubric directly distinguishes ambiguous guidance from model error.
The question
A financial-document classifier has substantial disagreement among annotators, especially for handwritten forms. Before approval, the team must determine whether disagreement reflects unclear labeling rules rather than model failure. Which evidence most directly resolves that uncertainty?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Compute overall accuracy on a larger labeled sample.A larger sample may improve precision, but it does not reveal whether annotators applied inconsistent or ambiguous labeling rules.
- Compare model outputs with last year’s production labels.Historical labels may reproduce the same unresolved disagreement and cannot independently establish which interpretation is correct.
- Adjudicate a stratified disagreement sample using an operational rubric. ✓Adjudication exposes disagreement causes, while stratification checks whether ambiguity concentrates in document types or subgroups.
- Run a privacy review of every document before measuring label agreement.Privacy review may be necessary for document handling, but it does not resolve whether labels were consistently interpreted.
The trap
When labels themselves are uncertain, inspect and adjudicate disputed examples before relying on model metrics. How to remember it
A stratified adjudication sample with an operational rubric directly distinguishes ambiguous guidance from model error.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Test re-identification risk and real-image subgroup: Before approval, which evidence is most important? →
- Obtain access to linked model: Which evidence should the team obtain? →
- Test targeted rare failure scenarios: Which evidence best addresses this risk before purchase? →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.
