Training Data Quality Checklist for Production ML

Q: What missing value rate in training data is too high to proceed?

A missing value rate above 5% in a key feature requires investigation and a documented imputation strategy before training. Below that threshold, standard imputation approaches are generally acceptable. The threshold applies per feature - a dataset can have low overall missingness but a critical predictor with concentrated missing values that will degrade model performance.

Q: How do I know if my training data labels are reliable enough?

Calculate inter-annotator agreement using Cohen's Kappa or Fleiss' Kappa across your annotators. A score below 0.7 indicates significant labelling inconsistency that will introduce noise into model training. Also estimate label noise rate using tools like Cleanlab - label noise above 10% typically requires relabelling or noise-robust training techniques before proceeding.

Q: Does training data need to be scanned for PII before model training?

Yes. PII in training data creates GDPR and CCPA compliance exposure - and potentially embeds personal data into model weights that can be extracted through adversarial queries. Every dataset used for production model training should be scanned for names, emails, phone numbers, health data, and other identifiers, with documented provenance and consent records where required.

Q: What is training-production distribution mismatch and why does it matter?

Distribution mismatch occurs when the inputs your model sees in production differ statistically from the training data it learned on. It is the most common cause of post-deployment model degradation - the model performed well on training data but encounters real-world patterns it was not trained to handle. Explicitly comparing training and production data distributions is a required pre-launch check.

Q: Why is train/test data split timing so important for ML data quality?

Splits must be created before any preprocessing or feature selection , not after. If you preprocess the full dataset first and then split, information from the test set leaks into training through preprocessing statistics - producing optimistic evaluation metrics that do not generalize to production. This leakage is one of the most common causes of models that benchmark well but underperform after deployment.

Key Takeaways

Inter-annotator agreement below 0.7 (Cohen's Kappa) indicates labeling inconsistency that will introduce noise into model training; label noise above 10% typically requires relabeling or noise-robust training techniques before proceeding.
Training-production distribution mismatch is the most common cause of post-deployment model degradation - the model performed well on training data but encounters real-world patterns it was not trained to handle.
Train/validation/test splits must be created before any preprocessing or feature selection, not after - preprocessing the full dataset first leaks test-set information into training and produces inflated evaluation metrics that do not generalize.
PII in training data creates GDPR and CCPA compliance exposure and can embed personal data into model weights extractable through adversarial queries - every production training dataset requires a documented PII scan.

Bad training data produces bad models - and bad models in production are expensive to diagnose, expensive to fix, and damaging to customer trust. Data quality problems are the root cause of most silent production ML failures, yet they are consistently the most underinvested part of the ML development process.

This checklist covers the 15 most critical data quality checks before you train a production model.

Completeness

Missing value rate - What percentage of rows have missing values in each feature? Missing values above 5% in a key feature require investigation and a documented imputation strategy.
Sample coverage - Does your dataset cover the full distribution of inputs your model will see in production? Gaps in coverage become gaps in model performance.
Temporal coverage - Does your dataset cover the time periods relevant to your prediction task? A fraud model trained on 2020 data may not reflect 2024 fraud patterns.

Labelling Quality

Inter-annotator agreement - If your labels were created by multiple annotators, what is the agreement rate (Cohen’s Kappa or Fleiss’ Kappa)? Below 0.7 indicates significant labelling inconsistency.
Label noise estimate - Use a confident learning technique (e.g., Cleanlab) to estimate the label noise rate. Label noise above 10% typically requires relabelling or noise-robust training.
Label distribution - Is your label distribution severely imbalanced? Class imbalance requires documented handling strategy (oversampling, class weights, threshold adjustment).
Annotation guidelines - Are annotation guidelines documented and version-controlled? Undocumented guidelines produce inconsistent labels over time.

Distribution

Feature distributions - Have you visualised the distribution of all key features? Unexpected spikes, bimodal distributions, or heavy tails require investigation.
Outlier identification - Have outliers been identified and handled (removed, winsorized, or kept with justification)? Undocumented outliers often indicate data quality issues upstream.
Training-production distribution match - Does your training data distribution match what your model will see in production? Distribution mismatch is the most common cause of model degradation post-deployment.

Privacy and Compliance

PII scan - Has the dataset been scanned for personally identifiable information (names, emails, phone numbers, SSNs, health data)? PII in training data creates GDPR/CCPA compliance exposure.
Consent and provenance - Is the data provenance documented? Is there evidence of consent for the intended ML use, where required?
Sensitive attribute handling - If the dataset includes sensitive attributes (race, gender, health status), is their inclusion documented and justified?

Data Management

Version control - Is the exact dataset version (including preprocessing steps) used for each model training run recorded? Without version control, you cannot reproduce a model or diagnose a regression.
Train/validation/test split - Are splits created before any preprocessing or feature selection? Leakage between splits produces optimistic evaluation metrics that don’t generalise to production.

How to Use This Checklist

Run this checklist before training your first production model version - and before any major retrain on new data. Document the outcome of each check and the actions taken. This documentation becomes part of your model card and is directly relevant to regulatory compliance and investor due diligence.

Book a data quality audit if you want an independent assessment of your training data against these criteria.

Common Questions

Frequently Asked Questions

What missing value rate in training data is too high to proceed?

A missing value rate above 5% in a key feature requires investigation and a documented imputation strategy before training. Below that threshold, standard imputation approaches are generally acceptable. The threshold applies per feature - a dataset can have low overall missingness but a critical predictor with concentrated missing values that will degrade model performance.

How do I know if my training data labels are reliable enough?

Calculate inter-annotator agreement using Cohen's Kappa or Fleiss' Kappa across your annotators. A score below 0.7 indicates significant labelling inconsistency that will introduce noise into model training. Also estimate label noise rate using tools like Cleanlab - label noise above 10% typically requires relabelling or noise-robust training techniques before proceeding.

Does training data need to be scanned for PII before model training?

Yes. PII in training data creates GDPR and CCPA compliance exposure - and potentially embeds personal data into model weights that can be extracted through adversarial queries. Every dataset used for production model training should be scanned for names, emails, phone numbers, health data, and other identifiers, with documented provenance and consent records where required.

What is training-production distribution mismatch and why does it matter?

Distribution mismatch occurs when the inputs your model sees in production differ statistically from the training data it learned on. It is the most common cause of post-deployment model degradation - the model performed well on training data but encounters real-world patterns it was not trained to handle. Explicitly comparing training and production data distributions is a required pre-launch check.

Why is train/test data split timing so important for ML data quality?

Splits must be created before any preprocessing or feature selection, not after. If you preprocess the full dataset first and then split, information from the test set leaks into training through preprocessing statistics - producing optimistic evaluation metrics that do not generalize to production. This leakage is one of the most common causes of models that benchmark well but underperform after deployment.