Estimate costs and capacity before comparing: Which next decision best supports a defensible threshold?
Threshold selection must reflect asymmetric error costs, reviewer capacity, subgroup variation, and uncertainty rather than accuracy alone.
The question
A research office uses a model to flag grant applications for manual review. False positives consume scarce reviewer hours; false negatives can exclude eligible projects from timely funding. The office has no validated cost estimates and performance differs across disciplines. Which next decision best supports a defensible threshold?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Optimize each discipline separately using observed rates, without validating the cost of errors.Separate optimization may improve measured rates but still cannot determine an acceptable tradeoff without validated costs and capacity assumptions.
- Select the most accurate threshold.Accuracy weights errors alike and ignores unequal consequences, reviewer capacity, and discipline-specific performance.
- Estimate costs and capacity before comparing discipline-specific thresholds. ✓Validated error costs, reviewer capacity, subgroup evidence, and uncertainty are needed to compare thresholds responsibly.
- Increase review volume to reduce missed applications.A lower threshold may reduce false negatives but could overwhelm reviewers, and its tradeoff is not yet supported by cost or capacity evidence.
The trap
For asymmetric errors, identify costs, capacity, subgroup effects, and uncertainty before choosing a threshold. How to remember it
Threshold selection must reflect asymmetric error costs, reviewer capacity, subgroup variation, and uncertainty rather than accuracy alone.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Set subgroup acceptance criteria and investigate: Before deployment, which evaluation decision is most →
- Conduct security testing for prompt injection: Which evidence gap should receive priority? →
- Measure agreement and adjudicate disputed labels using: What should happen before training proceeds? →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.
