Production ML pipeline with MLflow and Airflow
Taking a model from notebook to production: feature pipelines in Spark, experiment tracking and a model registry in MLflow, batch scoring orchestrated by Airflow, and drift monitoring.
The problem
Training a model is the easy part. Keeping it accurate, reproducible and observable in production is where most ML projects stall.
Scope
- Features: a Spark pipeline that builds point-in-time-correct features.
- Training: experiments tracked in MLflow, with the best model promoted through the registry.
- Scoring: an Airflow DAG that runs batch inference on a schedule, idempotently and with retries.
- Monitoring: data and prediction drift checks that alert when the model needs attention.
- CI/CD: tests and image builds on every pull request.
Milestones
- Feature pipeline with tests
- Training and registry workflow
- Airflow scoring DAG running locally with Docker Compose
- Drift monitoring and alerting
- Write-up: what “production-ready” actually required