Adjudicate sampled disagreements and revise the labeling: What should the team do first?
High disagreement signals unclear or potentially biased labels; adjudication and rubric revision should precede model training.
The question
A housing application classifier has 30% disagreement between two trained annotators on whether documents are complete. The model cannot proceed until labels are reliable, and the disagreement appears concentrated in accessibility-related formats. What should the team do first?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Adjudicate sampled disagreements and revise the labeling rubric. ✓Reviewing disagreements reveals ambiguity or bias, while rubric revision creates a more consistent basis for reliable subsequent annotation.
- Select the annotator whose labels better match historical decisions.Historical agreement may reproduce prior inconsistencies and does not establish that either annotator correctly handles accessibility-related formats.
- Train on both annotators’ labels and average their outputs.Averaging conflicting labels can conceal systematic disagreement and accessibility-related ambiguity instead of improving the underlying label definition.
- Remove all records where annotators disagree.Removing disagreements may discard precisely the accessibility-related cases needed to define complete-document criteria fairly.
The trap
Investigate concentrated disagreement before aggregating, deleting, or privileging labels. How to remember it
High disagreement signals unclear or potentially biased labels; adjudication and rubric revision should precede model training.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Test synthetic records for privacy leakage before use: Which action is most immediately necessary? →
- Version the model: Which documentation choice is most useful? →
- Build a targeted defect test set and require: Which evaluation change is most defensible? →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.
