A model that performs well in a Jupyter notebook and a model that performs well in production are not the same achievement. MLOps, the practices and tooling for deploying, monitoring, and maintaining ML models reliably, exists because the gap between the two is where most real-world ML projects actually fail. Gartner and various industry surveys have repeatedly found that a majority of ML models built are never deployed at all, and a meaningful share of the ones that are deployed get abandoned within a year, usually not because the model was inaccurate at launch, but because nobody had a plan for what happens after. This is precisely the gap Microsoft’s Frontier Company is betting enterprises will pay to have someone else close for them, rather than building this discipline in-house.
Google’s own MLOps engineering guidance frames the core problem the same way: the hard part was never training a single model, it’s building and continuously operating the integrated system around it — data pipelines, retraining triggers, evaluation gates, and serving infrastructure — since that system, not the one-off notebook result, is what actually has to survive contact with production (Practitioners Guide to MLOps, Google Cloud; MLOps: Continuous delivery and automation pipelines in machine learning, Google Cloud Architecture Center).
Model Versioning
The first problem MLOps solves is one that sounds trivial until you’ve lived through it: knowing exactly which model, trained on exactly which data, with exactly which code and hyperparameters, is currently serving predictions in production. Unlike traditional software, an ML model’s behavior depends on three things that all need to be tracked together: the code that trains it, the data it was trained on, and the resulting weights. Changing any one of the three produces a different model, and being unable to reproduce a specific past model (to debug a regression, satisfy an audit, or roll back a bad deploy) is a common and avoidable failure mode.
Three widely used open-source tools illustrate the three angles teams typically need covered:
- MLflow’s Model Registry provides a centralized store with versioning, lineage (which experiment/run produced a given model), and stage transitions (staging → production → archived), so “which model is live” has one authoritative answer instead of living in a spreadsheet or someone’s memory (MLflow Model Registry, official documentation).
- DVC (Data Version Control) extends Git-style versioning to data and model artifacts specifically, and its experiment-tracking workflow automatically records the data dependencies, code, parameters, and metrics behind each run so a specific past result can actually be reproduced, not just described (Experiment Tracking, official DVC documentation).
- Weights & Biases focuses on the observability side of the same problem: logging hyperparameters, metrics, and artifacts for every run into an interactive dashboard so runs can be compared and regressions traced back to a specific change (Experiments overview, official Weights & Biases documentation).
These tools are complementary rather than competing, in practice, many production teams run a registry (MLflow) alongside a dedicated experiment tracker (W&B) and data/artifact versioning (DVC) rather than expecting any single tool to cover all three jobs.
Continuous Training and Model Drift
A model trained once and deployed forever degrades, because the world it was trained on keeps changing while its weights stay frozen. This is called drift, and it comes in two flavors: data drift, where the distribution of incoming inputs shifts (user behavior changes, a new product category appears, a sensor gets recalibrated), and concept drift, where the actual relationship between inputs and the correct output changes (fraud patterns evolve specifically to evade the current fraud model, consumer preferences shift).
Continuous training pipelines address this by automating retraining on fresh data on a schedule or trigger, rather than treating training as a one-time event before launch. The harder engineering problem isn’t the retraining itself, it’s building an evaluation gate that automatically checks a freshly retrained model against the current production model on held-out data before it’s allowed to replace it, so a bad batch of training data doesn’t silently degrade a system that was working fine.
Monitoring and Alerting
Once a model is live, the question shifts from “is it accurate” to “how would we know if it stopped being accurate.” This is harder than it sounds for one specific reason: in most production systems, you don’t get the ground-truth correct answer at prediction time, you find out later, if ever. A recommendation model finds out days later whether a user actually engaged; a fraud model might not get confirmed ground truth for weeks pending investigation.
Production ML monitoring therefore leans heavily on proxy signals that don’t require waiting for ground truth: tracking the statistical distribution of input features and prediction outputs for sudden shifts (a strong signal something upstream broke, even before accuracy numbers are available), tracking prediction confidence/calibration over time, and setting alerts on infrastructure-level signals like latency and error rate that indicate the serving system itself is unhealthy, separately from whether the model’s judgments are still good. Mature MLOps setups pair this with a feedback loop that captures ground truth as it eventually becomes available and feeds it back into both evaluation dashboards and the next retraining cycle.
Testing an ML Pipeline, Not Just an ML Model
Traditional software testing (unit tests, integration tests) doesn’t disappear in an ML system, but it isn’t sufficient on its own, because a pipeline can pass every conventional test and still ship a bad model if the data going into it is broken. Google’s own internal ML engineering guidance, published externally as “Rules of Machine Learning,” makes this point directly: it recommends treating pipeline health as testable and monitorable in the same way as any other production software, with explicit checks on things like whether a feature’s data still resembles what the model was trained on, rather than only testing the code that implements the pipeline (Rules of Machine Learning: Best Practices for ML Engineering, Google Developers). In practice, a reasonably mature ML testing setup has at least three distinct layers:
- Data validation tests: schema checks (are the expected columns present, in the expected types and ranges) run automatically on every new batch of training or inference data, catching a broken upstream data pipeline before it silently retrains a model on garbage.
- Model quality tests: the evaluation gate described above — a candidate model must beat the current production model (or a fixed quality floor) on a held-out set before it’s allowed to deploy, not just “finish training without crashing.”
- Infrastructure/serving tests: standard software tests for the serving layer itself (latency, throughput, correct request/response shape) that would exist for any production service, ML-powered or not.
Why This Matters More as Models Get Bigger
The rise of large foundation models has shifted, but not eliminated, the MLOps problem. Fewer teams are training models from scratch, but far more are fine-tuning, prompt-engineering, and building retrieval pipelines on top of foundation models, and all the same failure modes apply: which base model version and which fine-tune/prompt configuration is live, how do you detect when the underlying hosted model provider silently updates their model in a way that changes your application’s behavior, and how do you catch quality regressions in generated output when there’s no single “ground truth” to compare against the way there was for a classifier. LLM-specific evaluation tooling (using another model as a judge, tracking hallucination rate on domain-specific test sets, structured prompt regression testing) has emerged largely to fill this same role in a generative context.
MLOps Tooling by Lifecycle Stage
The tooling landscape maps fairly cleanly onto the stages above, which is useful when deciding what to actually adopt first rather than trying to stand up an entire platform at once:
| Lifecycle Stage | What It Solves | Representative Tools |
|---|---|---|
| Experiment tracking | Recording hyperparameters, metrics, and results per run for comparison | Weights & Biases, MLflow Tracking |
| Data & artifact versioning | Reproducing a specific past model exactly, including its training data | DVC, MLflow artifacts |
| Model registry | One authoritative answer to “which model version is live” | MLflow Model Registry |
| Orchestration / continuous training | Automating retraining on a schedule or trigger, with gated promotion | Kubeflow Pipelines, Google Cloud Vertex AI Pipelines |
| Production monitoring | Detecting drift and system health without waiting on ground truth | Custom dashboards on top of the metrics stores above |
The Practical Baseline
For a team standing up ML in production for the first time, the minimum viable version of all this is: version your training data and model artifacts together, not just your code; have an automated evaluation step that runs before any new model replaces the current one; and monitor input/output distributions in production, not just an accuracy number measured once at deployment time. Everything more sophisticated than that, a full registry, orchestrated continuous training, dedicated drift dashboards, is an optimization on top of a foundation that, without these three things, tends to fail in ways that are expensive and slow to diagnose.
