Run stratified tests on multilingual workflow samples: Which evidence best resolves the uncertainty?
Use stratified tests in the real workflow because aggregate and vendor benchmarks cannot answer the multilingual subgroup question.
The question
A bank is red-teaming a financial-document model before release. Testing found strong aggregate accuracy, but reviewers cannot determine whether prompt-based extraction failures disproportionately affect multilingual applicants; the vendor also supplied only benchmark results. Approval requires evidence tied to the deployed workflow and subgroup uncertainty. Which evidence best resolves the uncertainty?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Obtain supplier subgroup results from comparable deployments and review their methods.Comparable supplier results may provide context, but they do not directly establish performance on the bank’s documents, prompts, and acceptance criteria.
- Have reviewers inspect multilingual outputs and document their overall impressions alongside the benchmark.Qualitative review can identify examples, but impressions alone do not provide systematic subgroup measurements or defined approval criteria.
- Expand the held-out aggregate benchmark sample.A larger aggregate sample improves overall estimates but does not isolate multilingual outcomes or reproduce the bank’s workflow.
- Run stratified tests on multilingual workflow samples. ✓Stratified testing in the deployed workflow directly measures the suspected disparity and can apply predefined subgroup acceptance criteria.
The trap
Match the evidence design to the suspected failure mode, subgroup, and deployment context. How to remember it
Use stratified tests in the real workflow because aggregate and vendor benchmarks cannot answer the multilingual subgroup question.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Revalidate the changed version: What is required before approval? →
- Reproduce the disclosure across controlled model: Which evidence should be prioritized? →
- Test rollback with current records: Which evidence is most important? →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.
