Compare model outcomes with and without distance: Which analysis is most informative?
Compare outcomes with and without the proxy, while examining subgroup results.
The question
A school district uses commute distance to prioritize transportation assistance. Distance is strongly associated with neighborhood income, but income is not in the training data. Before approval, the district must assess whether distance creates unequal results without collecting income from every applicant. Which analysis is most informative?
Preparing for AIGP? Take the free 5-min readiness quiz →
- Stratify decisions by neighborhoodNeighborhood stratification can reveal outcome differences but does not isolate whether commute distance is driving them.
- Collect income from every applicant before testingIncome could support additional analysis, but universal collection is not necessary for the stated initial proxy investigation and raises minimization concerns under GDPR where applicable.
- Compare districtwide prioritization accuracyAn aggregate accuracy measure can hide neighborhood disparities and does not isolate the proxy’s effect.
- Compare model outcomes with and without distance ✓A controlled feature comparison tests distance’s contribution while allowing subgroup outcome analysis without requiring universal income collection.
The trap
For a suspected proxy, isolate the feature’s effect and inspect subgroup outcomes; do not rely only on aggregate accuracy. How to remember it
Compare outcomes with and without the proxy, while examining subgroup results.
How many of these would you get right?
One of 1581 AIGP questions on Certsqill. Take a free five-minute check and see your score per domain — not one number, but which section to open tonight.
Test your AIGP readiness — freeMore Understanding How to Govern AI Development questions
- Compare labels with independently assessed service needs: What should reviewers examine first? →
- Evaluate once on a held-out December-like set: Which evidence would most directly address whether performance →
- Create a deduplicated held-out clip set: Which evidence should reviewers request? →
- All 426 Understanding How to Govern AI Development questions →
Part of the Certsqill AIGP question bank · Understanding How to Govern AI Development ·
Every answer, right and wrong, comes with its own explanation.