Remove the leaking feature: Which course of action best reflects sound management of issues and risks during

Data leakage invalidates the results, so the fix is to remove it, retrain, re-evaluate, and document the impact.

The question

Preparing for AIGP? Take the free 5-min readiness quiz →

1. Proceed to launch on the current scores but schedule a post-launch fix, since the fixed deadline cannot slip now.

   Almost right to plan a fix, but launching on scores known to be invalid ships an unmanaged, hidden risk.
2. Keep the feature but discount the reported metrics by a fixed margin to approximate its inflating effect on scores.

   Almost right to distrust the numbers, but a guessed discount neither removes leakage nor yields valid evaluation.
3. Remove the leaking feature, retrain and re-evaluate, and document the issue and its impact on prior results. ✓

   Leakage invalidates results, so it must be fixed, re-evaluated, and documented before any reliance on scores.
4. Report the strong validation scores as-is and note the leakage only in an internal engineering ticket.

   Plausible-sounding but presenting known-inflated scores as valid misrepresents the model's true performance.

How to remember it

Data leakage invalidates the results, so the fix is to remove it, retrain, re-evaluate, and document the impact.

More Understanding How to Govern AI Development questions

• Interpretability testing to surface which features drive: Given the goal of understanding the reasoning behind →
• Record the datasets: Which improvement most directly satisfies the documentation objective for training and →
• Confirming the system meets its performance: Which outcome best reflects the purpose of that assessment? →
• All 426 Understanding How to Govern AI Development questions →