Establish adjudication and measure annotator agreement: Which missing control is most direct?
Frequent annotation disagreement requires adjudication and agreement measurement before treating labels as dependable training evidence.
The question
A logistics scheduling model uses human labels for delivery-delay causes. Accuracy testing, subgroup coverage, provenance, privacy controls, and deployment monitoring are complete. Two trained annotators disagree frequently on borderline cases, but no adjudication rule or agreement measure exists. Which missing control is most direct?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Report only overall scheduling accuracy after deployment.Overall accuracy can conceal label ambiguity and subgroup or case-specific weaknesses; it does not establish the quality of disputed annotations.
- Establish adjudication and measure annotator agreement. ✓Adjudication resolves disagreements, while agreement measurement quantifies label consistency and identifies whether training labels are sufficiently reliable.
- Add more unlabeled delivery records.More unlabeled records increase volume without resolving inconsistent judgments or establishing whether existing labels reliably represent delay causes.
- Increase model capacity for borderline cases.Greater model capacity may fit ambiguous labels more closely, but it cannot determine which conflicting annotation reflects the intended outcome.
The trap
When labels conflict, improve the labeling process before optimizing the model. How to remember it
Frequent annotation disagreement requires adjudication and agreement measurement before treating labels as dependable training evidence.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Perform privacy-leakage and re-identification testing: Which control is most direct? →
- Record model-data version identifiers: What is the most direct missing control? →
- Create targeted tests for the rare severe condition: Which additional control most directly addresses this →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.
