MLOps pipeline: Stages, tools, and deployment workflow

Machine learning workflows require orchestrating complex dependencies across data ingestion, feature engineering, training, evaluation, and deployment stages. Manual execution of these interdependent steps doesn't scale beyond small teams, leading to inconsistent results, long development cycles, and difficulty reproducing successful experiments.

Automated MLOps pipelines solve this by modeling each stage as an orchestrated workflow with explicit dependencies, handoffs, and failure handling. The result is reproducible model development and deployment that holds up at scale. Raw data enters a pipeline that moves through ingestion and validation, feature engineering, distributed training, evaluation gates, and finally production deployment, where monitoring detects drift and triggers retraining. This article examines each stage and the orchestration tooling that manages handoffs between them.

Data ingestion and validation pipeline orchestration

Production ML systems consume data from heterogeneous sources operating on different schedules and reliability profiles. For example, the fraud detection system may pull transaction records from PostgreSQL databases every hour, enrichment data from third-party APIs on daily schedules, and real-time event streams from Kafka topics. Orchestration platforms like Airflow manage these dependencies so downstream feature pipelines only start once all required sources have delivered their data.

Incoming data then passes through a validation layer that runs schema conformance checks, value-range constraints, statistical distribution checks, and duplicate detection. Clean batches are routed to feature engineering, while failed batches are quarantined, and alerts are sent to the data engineering team.

The diagram below illustrates an example of such a flow. PostgreSQL, third-party APIs, and Kafka streams feed into Airflow orchestration, which routes data through Great Expectations validation. The validation layer runs schema conformance checks, value-range constraints, statistical distribution checks, and duplicate detection before directing clean batches to feature engineering or quarantining failed batches for investigation and alerts.

An example flow for data ingestion and validation performed by Airflow, with incoming data routed to feature engineering or quarantine.

Automated quality gates and schema validation

Data quality issues can propagate through ML pipelines, producing models trained on corrupted inputs. Great Expectations implements automated validation, schema conformance, value-range constraints, and statistical distribution shifts. When transaction amounts suddenly shift from typical ranges or categorical fields contain unexpected values, validation failures halt pipeline execution and trigger alerts to data engineering teams.

The validation layer prevents insufficient or malformed data from reaching the training stages. Quality checks verify record counts meet minimum thresholds, confirm temporal coverage spans expected date ranges, and detect duplicate entries that could bias model learning. Orchestration platforms route clean data to feature engineering while quarantining failed batches for investigation.

Incremental processing and change data capture

Incremental loading processes only changed records rather than reprocessing entire datasets, dramatically reducing computational requirements and pipeline execution time. Change data capture (CDC) mechanisms track modifications in source databases, allowing pipelines to update feature stores with new transactions while preserving historical feature values for training reproducibility.

Lineage tracking maintains relationships between raw data sources, intermediate transformations, and final training datasets. When data quality issues surface in production models, teams trace lineage graphs backward to identify which source data corrupted feature calculations, enabling targeted remediation without re-executing the entire pipeline.

Feature engineering and transformation pipeline architecture

Feature pipelines transform raw transaction data through a layered dependency graph. The sequencing matters: First compute individual transaction features (amount normalization, timestamp encoding), then aggregate features over time windows (7-day spending velocity, merchant frequency), then derive cross-feature interactions (velocity vs merchant risk score). Each layer depends on the one below it, so orchestration has to enforce execution order before any training job starts.

Here is where things get complicated. Those same features need to serve two very different execution environments: Offline training runs as high-throughput Spark batch jobs processing historical data, while online inference runs in low-latency microservices handling individual requests in real time. Without a shared transformation layer, subtle implementation differences accumulate into training-serving skew, where models perform well offline but degrade in production. A feature store like Feast or Tecton seeks to solve this by acting as a single transformation definition, providing the training pipeline with consistent historical data access and the application model with low-latency, real-time serving from the same source.

Preventing training-serving skew

The most insidious pipeline failure occurs when feature computation differs between training and inference environments. Offline training typically processes historical data using Spark batch jobs running in data warehouses, while online inference requires real-time feature computation in low-latency microservices. Keeping that feature logic consistent across two such different execution environments is harder than it sounds.

Training-serving skew causes accuracy degradation when subtle implementation differences accumulate. A date parsing function with different timezone handling, floating-point precision differences between Spark and Python inference code, or missing data imputation strategies that diverge across systems all introduce prediction errors that go undetected during offline evaluation. Production deployments experiencing this skew can show meaningful degradation in accuracy compared with offline metrics.

Feature stores like Feast and Tecton solve this by using a single transformation definition that serves both training and inference. Teams can write feature transformations once, and the feature store automatically materializes them to offline stores for training batch jobs and to online stores for real-time inference requests. This architectural pattern seeks to prevent implementation drift.

The example below illustrates how this works in practice using Feast. A single @feature_view decorator marks the transformation as available for both offline and online use. The ttl=timedelta(days=30) parameter tells Feast how far back to look when materializing historical data for training. Setting both online=True and offline=True means the same transaction_velocity function computes 7-day transaction counts and 30-day spend totals regardless of whether the request comes from a batch training job or a real-time inference call.

Orchestrating feature dependency graphs

Pipeline orchestration manages execution ordering when features depend on other feature computations. Merchant risk scores aggregate historical fraud patterns, then transaction-level features multiply merchant risk by transaction amount. Directed acyclic graphs (DAGs) in orchestration platforms sequence these dependencies, parallelizing independent computations while respecting prerequisite relationships.

Feature monitoring detects distribution shifts and performance degradation before they corrupt model predictions. Tracking compute latency identifies when feature calculations exceed SLA thresholds, while statistical profiling catches data drift in feature distributions. Automated alerts trigger when cache miss rates spike or feature freshness degrades, preventing stale features from reaching inference endpoints.

Model training and hyperparameter optimization workflow orchestration

Training pipelines coordinate multiple moving parts simultaneously. When a training job starts, the pipeline has to load features from the feature store, allocate GPU resources, launch distributed workers, track experiment metrics, and save checkpoints in case of failure. Miss any of these coordination points, and you either waste expensive compute or lose experiment reproducibility. Kubeflow Pipelines and Vertex AI Pipelines handle this by modeling the entire workflow as a DAG where each step runs in an isolated container with explicit input/output dependencies.

The diagram below shows three parallel workflow tracks that Kubeflow or Vertex AI can manage simultaneously. The top track handles distributed training across multiple A100/H100 GPUs, running data loading, preprocessing, training, experiment tracking, and evaluation across parallel workers. The middle track runs hyperparameter optimization using tools like Optuna or Ray Tune, where each trial executes the same sequence independently, with checkpoint management preserving trial state so failed trials can resume without restarting from scratch. The bottom track feeds every run into MLflow, capturing metrics, parameters, and artifacts from each trial so teams can compare results across experiments.

Model training and hyperparameter optimization workflow covering distributed training, parallel trials, and experiment tracking via Kubeflow Pipelines or Vertex AI.

On the right side of the diagram, three outputs come out of this orchestration layer: automatic retry logic for infrastructure failures, resource quotas that prevent training jobs from consuming the entire cluster, and metrics that are captured and fed to the model registry downstream. Together, these mechanisms mean a training pipeline can run dozens of hyperparameter trials overnight, recover from spot instance preemptions, and surface the best-performing checkpoint to the evaluation stage without manual intervention.

Distributed training and resource management

Modern neural network architectures require distributed training across multiple GPUs to achieve reasonable training times. A fraud detection model using transformer architectures might distribute batch processing across eight A100 or H100 GPUs (as newer hardware becomes available), with orchestration platforms managing resource allocation and monitoring job health.

Pipeline orchestration handles training failures through automatic retry logic. When spot instances terminate mid-training or CUDA out-of-memory errors occur, checkpointing mechanisms save intermediate model states, allowing resumed execution from the last successful checkpoint rather than restarting from scratch. For multi-day training runs on expensive GPU infrastructure, this fault tolerance is what makes the difference between a recoverable failure and a complete restart.

Resource quotas prevent runaway experiments from consuming entire GPU clusters. Orchestration platforms enforce limits on simultaneous training jobs, per-user GPU allocation, and maximum job duration. This keeps resource sharing fair across teams and prevents a single runaway experiment from taking down the cluster.

Hyperparameter optimization at scale

Hyperparameter optimization (HPO) requires running dozens or hundreds of training runs to test different configurations. Frameworks like Optuna and Ray Tune integrate with pipeline orchestration to manage trial scheduling, early stopping of underperforming runs, and GPU resource sharing across concurrent trials.

The Kubeflow configuration below shows how this gets defined in practice.

• The algorithmName: bayesian setting tells Kubeflow to use Bayesian optimization rather than random search, meaning each new trial uses results from previous trials to pick more promising configurations.
• Setting parallelTrialCount: 4 runs four trials simultaneously, balancing GPU utilization against the search strategy's need for sequential feedback.
• The maxTrialCount: 50 caps the total search budget.
• The two-parameter blocks define the search space: learning_rate searches between 0.0001 and 0.1, and dropout_rate searches between 0.1 and 0.5.

Kubeflow samples values from these ranges using a Bayesian algorithm, progressively narrowing toward configurations that perform well.

Experiment tracking captures hyperparameters, metrics, artifacts, and environment details for every training run. MLflow, Weights & Biases, or Neptune log training curves, final model checkpoints, and infrastructure metadata, enabling teams to reproduce experiments months later and compare performance across hundreds of trials.

Model evaluation and validation gate implementation

The evaluation stage is where you determine whether a trained model is ready for production. The pipeline runs three parallel assessment tracks against a holdout dataset: accuracy testing (F1 score, AUC-ROC, RMSE), fairness evaluation (checking demographic parity across user segments), and performance validation (measuring p95 latency under realistic load). Running these in parallel reduces evaluation time while covering the dimensions that matter for production readiness.

The diagram shows what happens after those three tracks are complete. Results feed into champion-challenger testing, where the new model variant is compared directly against the current production champion using consistent comparison metrics. This comparison drives a decision gate with two outcomes. If the new model passes, it moves into the MLflow Model Registry, progressing through development and staging environments before reaching production. Manual sign-off is required before the final promotion step, giving teams a checkpoint before any model reaches live traffic.

Model evaluation and validation gate with champion-challenger comparison, MLflow Registry promotion, and retraining loop for failed models.

If the model fails the decision gate, it routes to investigation and retraining. This is not just a rejection but a structured feedback loop. The pipeline captures root cause analysis, identifies whether data augmentation or hyperparameter tuning can address the failure, and triggers retraining with those adjustments. The retrained model then re-enters the evaluation pipeline from the top, running through the same accuracy, fairness, and performance checks before facing the decision gate again.

Evaluation approaches for different model types

LLM applications require different evaluation approaches beyond traditional metrics. AgentControl includes Online Evaluations using LLM-as-judge techniques for measuring accuracy, relevance, factual consistency, and toxicity that standard metrics cannot capture. These evaluations run automatically during inference, providing continuous quality assessment rather than point-in-time testing.

For example, a customer support chatbot evaluates response quality by having a separate LLM judge whether answers are helpful, empathetic, and factually accurate, metrics invisible to traditional accuracy measurements but critical for user satisfaction.

Champion-challenger testing workflows

Production ML systems implement champion-challenger comparison, requiring new model variants to demonstrate measurable improvement before replacing deployed models. The setup is straightforward: The current production model is the champion, and the new candidate is the challenger. Both run in parallel against the same test sets through A/B testing, so the comparison reflects identical conditions rather than different evaluation environments.

For traditional ML models, these gates translate to concrete requirements: accuracy gains alongside latency reductions across multiple independent test sets. LLM applications use different comparison metrics, including user satisfaction, hallucination rates, and response quality, measured through automated LLM-as-judge evaluations or human feedback. A new prompt variant might show improved factuality scores and reduced hallucination rates while staying within acceptable generation latency and token costs, which is exactly the kind of multi-dimensional tradeoff these validation gates are designed to surface.

Model registry integration and approval workflows

The MLflow Model Registry provides staging workflows implementing development, staging, and production promotion gates. Models that pass offline evaluation enter staging environments, where they serve shadow traffic against real production requests without affecting any actual predictions.

Approval workflows require manual sign-off from data science and engineering teams before production deployment. Registry lineage tracking links each deployed model back to its training dataset, feature store version, and hyperparameters. Later on, when a production incident occurs, that lineage is exactly how teams trace the root cause.

Production deployment, orchestration, and automated feedback loops

Deployment pipelines promote validated models from registries to serving infrastructure like KServe, SageMaker, or Kubernetes-based inference endpoints. However, traditional deployment automation ends at infrastructure availability without controlling user exposure or measuring business impact. Coordinating gradual rollout, A/B testing, and rollback requires integration beyond standard CI/CD pipelines.

Progressive rollout and production experimentation

Testing new ML model versions against production traffic requires either maintaining parallel inference infrastructure or building custom traffic-routing solutions. Blue-green deployments can increase inference costs during testing periods, while manual canary deployments can require additional traffic management and monitoring. These approaches create operational complexity and scale poorly for organizations testing multiple model variants simultaneously.

Consider a fraud detection team running two model versions in parallel: the current production model and a challenger trained on more recent transaction patterns. They want to test the challenger against North American traffic first before rolling it out globally, while keeping European traffic on the current model until regional validation completes. Blue-green deployment can mean running parallel inference environments during testing; a manual canary setup can mean hand-adjusting traffic splits and correlating metrics across segments. Add a third variant for a high-value cohort and the coordination overhead compounds quickly.

LaunchDarkly feature flags (CodeControl) target this problem by decoupling model selection from deployment. Instead of routing traffic at the infrastructure layer, the serving code evaluates a flag that returns which model version to use for each request. Targeting rules, user attributes, geographic region, or a custom cohort decide the split, so teams can run the current model and a challenger side by side without standing up parallel clusters or modifying application code.

LaunchDarkly ai-tooling repo includes skills that can scaffold the first parts of this workflow, including creating the model-version flag, targeting users by region or segment, and setting percentage rollouts.

LaunchDarkly supports both progressive rollouts and guarded rollouts. For example, a team might use a progressive rollout to gradually increase exposure to a new recommendation model. When metric-based regression monitoring is required, the team can instead configure a guarded rollout and enable automatic rollback if LaunchDarkly detects a regression. The flag handles segmentation, so no custom routing logic lives in the model serving layer.

Connecting a model serving layer to LaunchDarkly feature flags requires one runtime flag evaluation: build a context from the request, evaluate the model-version flag, and load whichever model version the flag returns.

To measure the rollout, attach a metric (model accuracy, error rate, p95 latency, or a business metric such as fraud loss) to the flag. A guarded rollout can monitor selected metrics and, when automatic rollback is enabled, revert the release if LaunchDarkly detects a regression. An experiment compares variations using a configured statistical methodology, with results interpreted based on factors such as the selected metric, sample size, experiment design, analysis method, and decision criteria.

The same repo also includes skills for configuring guarded rollouts, choosing the right rollout metric, creating that metric, and instrumenting metric events from the application.

A/B testing business and technical metrics

Production experiments measure business impact beyond model accuracy metrics. While offline evaluation may show that a new fraud model achieves higher F1 scores, production A/B tests measure whether improved accuracy translates into reduced fraud losses, acceptable false-positive rates affecting legitimate customers, and revenue impact from declined transactions. Once your test is running, you must track whether the new model improves baseline performance. You can use LaunchDarkly's AI tooling to choose, create, and instrument the right metrics, and then automatically run the A/B experiment between versions.

For traditional ML model rollouts, CodeControl helps teams compare model versions using metrics such as latency, error rate, conversion, false-positive rate, or revenue impact. For LLM applications, AgentControl applies when teams need to evaluate prompt variants, model parameters, response quality, hallucination rates, token cost, or LLM-as-judge results.

For experiment workflows, the repo includes a skill for setting up an A/B experiment between model versions, helping teams connect model-version rollout with statistical comparison.

Shadow deployment and production testing

Validating ML models against production traffic distributions requires expensive staging environments that attempt to replicate production data characteristics. These environments are expensive to maintain and never perfectly match production patterns. Temporal drift and sampling biases mean staged data is always an approximation.

Modern MLOps can reduce that overhead by running controlled tests against real traffic. LaunchDarkly CodeControl lets teams deploy new model versions behind flags, routing predictions to internal users or a small percentage of real traffic without affecting user experience. Models get validated against actual data distributions, not synthetic approximations.

Rollback and incident response

Model performance degradation from data drift, concept drift, or infrastructure issues requires emergency rollback capabilities. Traditional deployment revisions involve code commits, container rebuilds, and infrastructure updates. During an active incident affecting customer transactions, those additional deployment steps can delay recovery.

CodeControl’s feature flags model rollback without touching infrastructure. When accuracy metrics degrade or inference latency exceeds designated SLA thresholds, a LaunchDarkly configuration update can route traffic back to the previous model version without requiring a new application deployment. LaunchDarkly maintains an audit trail of configuration changes with timestamps and user attribution.

Automated remediation extends this pattern. When monitoring detects model drift or accuracy drops below designated thresholds, alerting systems can trigger a feature flag update to roll back to the previous model version and initiate a retraining workflow, such as a scheduled retraining DAG, a drift-triggered training job, or retraining after new labeled data becomes available.

Automated retraining feedback loops

Production monitoring creates feedback loops where drift detection and accuracy degradation automatically trigger retraining workflows. Scheduled refreshes (weekly or monthly, depending on data velocity) maintain model relevance. Event-driven retraining responds to distribution shifts or performance drops detected by monitoring systems.

For LLM applications, AgentControl captures user feedback through thumbs-up/down signals and quality scores via Online Evaluations. Those signals drive prompt iteration without requiring full model retraining. Teams correlate production metrics with specific prompt or model changes through the complete audit trail, which records every configuration change.

Orchestration platforms like Airflow implement these feedback loops through sensors that monitor model performance metrics and trigger DAG execution when retraining thresholds are met. The fraud detection pipeline might trigger retraining when false-positive rates exceed acceptable levels or when transaction pattern distributions shift beyond expected ranges.

Bridging ML training and application deployment

ML teams struggle to integrate model training outputs with application deployment pipelines. Coordinating model registry updates, feature store synchronization, and rollout orchestration manually creates deployment delays and rollback complexity. CodeControl integrates directly into CI/CD workflows and automates those coordination points.

LaunchDarkly's API lets CircleCI, GitHub Actions, or Azure DevOps pipelines programmatically create model version flags when training jobs complete. Canary rollouts start automatically once deployment validation passes. That eliminates manual handoffs between ML and DevOps teams and keeps deployment processes consistent across all model updates.

After the winning model is fully shipped, the repo’s flag cleanup skill can help teams retire temporary rollout flags so model-release controls do not become long-term technical debt.

CodeControl supports shadow deployments, multi-armed bandits, and geographic targeting. Multi-armed bandit algorithms can automatically adjust traffic allocation toward better-performing variants based on ongoing metric collection, selecting the best-performing model without manual intervention.

Conclusion

MLOps pipelines orchestrate complex workflows spanning data ingestion through production deployment. Each stage requires specialized tools and integration points. Data validation prevents training on corrupted inputs, feature stores reduce training-serving skew, distributed training orchestration manages expensive GPU resources, and validation gates block low-quality models from reaching production.

Production deployment is more than infrastructure availability. It requires controlled experimentation, progressive rollout automation, production metrics, and fast rollback. For traditional ML models, LaunchDarkly feature flags and CodeControl help teams control model-version rollout. For LLM applications, AgentControl supports prompt versioning, model parameter changes, online evaluations, and feedback loops.

The underlying principle is the same, replace manual coordination between teams with explicit pipeline contracts, automated gates, and observable handoffs. That applies whether you are shipping a gradient-boosted classifier or a prompt-driven LLM application.