MLOps — running models in production

Use virtual environments (venv, conda, or poetry) to isolate project dependencies. Pin exact versions in requirements.txt or pyproject.toml for reproducibility. Use conda for complex native dependencies like CUDA. Docker containers ensure full environment reproducibility. Tools like pip-tools or poetry provide dependency resolution and lock files.

MLOps (Machine Learning Operations) applies DevOps principles to ML systems, covering the full lifecycle from data preparation through model deployment and monitoring. It is important because most ML models fail to reach production without proper engineering practices. MLOps ensures reproducibility, reliability, scalability, and continuous improvement of ML systems in production environments.

An ML pipeline includes data ingestion and validation, feature engineering and storage, model training and hyperparameter tuning, model evaluation and validation, model registry and versioning, deployment (serving infrastructure), monitoring and alerting, and feedback loops for retraining. Each component should be automated, reproducible, and version-controlled.

Batch inference processes large volumes of data periodically (hourly, daily) and stores results for later use. Real-time inference processes individual requests with low latency (milliseconds). Batch is simpler and cheaper for recommendations or scoring; real-time is needed for fraud detection, search ranking, and conversational AI. Many systems use both: batch for pre-computation and real-time for personalisation.

Experiment tracking records hyperparameters, metrics, code versions, data versions, and artifacts for each training run. It enables comparison across experiments and reproducibility. Popular tools include MLflow (open-source, self-hosted), Weights & Biases (cloud-based with rich visualisation), Neptune, and Comet. Choose based on team size, budget, and integration requirements.

Model deployment is the process of making a model available in a production environment, including infrastructure setup, configuration, and release management. Model serving is the runtime system that handles inference requests, including request batching, model loading, scaling, and response formatting. Deployment is a one-time event; serving is the ongoing operational system.

Docker containers package the model, code, dependencies, and runtime into a portable, reproducible unit. This eliminates 'works on my machine' issues, ensures consistent behaviour across environments, simplifies deployment, and enables scaling via orchestrators like Kubernetes. Use multi-stage builds to minimise image size and separate training from serving containers.

AutoML automates the process of selecting models, tuning hyperparameters, and engineering features. Tools like Google AutoML, H2O, and Auto-sklearn make ML accessible to non-experts and accelerate prototyping. However, domain expertise and understanding of results remain essential for production deployments.

Test each transformation function with known inputs and expected outputs. Test edge cases like empty DataFrames, missing values, and unexpected types. Use pytest fixtures for reusable test data. Test the full pipeline end-to-end with a small sample. Verify output shapes, dtypes, and value ranges. Use hypothesis library for property-based testing of transformations.

Pickle serialises Python objects to byte streams for storage and loading. It is commonly used to save scikit-learn models and data. However, pickle can execute arbitrary code during deserialisation, making it a security risk for untrusted sources. Use safer alternatives like ONNX, safetensors, or joblib with known-source files. Never unpickle data from untrusted origins.

Use FastAPI or Flask to create endpoints. Load the model at startup (not per request). Define Pydantic models for request/response validation. Implement async handlers for IO-bound operations. Add health checks, logging, and error handling. Use gunicorn/uvicorn with multiple workers for production. Containerise with Docker and add rate limiting.

Prompt templates are parameterised prompts with placeholders filled at runtime (e.g., 'Summarise this {document_type} in {language}'). In production, manage them as versioned configuration with template engines like Jinja2. Store in a prompt registry, track performance per version, and implement rollback capability. LangChain and similar frameworks provide prompt template abstractions.

Implement incremental indexing that detects changed documents via checksums or timestamps. Use metadata to track document versions and update timestamps. For critical applications, maintain version history and allow querying against specific versions. Implement a document deletion pipeline that removes stale vectors. Schedule periodic full re-indexing for consistency.

Monitor retrieval latency, embedding computation time, and end-to-end response time. Track retrieval hit rate and relevance scores. Measure answer faithfulness and hallucination rate. Monitor index size growth and query volume. Track user feedback (thumbs up/down, corrections). Alert on unusual query patterns, error rates, or degradation in response quality.

Write unit tests for individual tool handlers with mock inputs. Use the MCP Inspector tool for interactive testing. Test with actual MCP clients like Claude Desktop. Verify schema validation for all inputs. Test error handling with invalid parameters. Load test for performance under concurrent connections. Test transport layer reliability with connection drops and reconnects.

Use the protocol version field in the initialise handshake. Add new tools without removing old ones. Deprecate tools by updating their descriptions before removal. Make new parameters optional with defaults. Use semantic versioning for the server itself. Test new versions against existing clients. Document breaking changes clearly. Support multiple versions simultaneously during transitions.

Use MCP's built-in logging capability to send structured log messages to the client. Log all tool invocations with parameters, execution time, and outcomes. Implement correlation IDs to trace requests across the system. Export metrics (latency, error rate, usage) to monitoring systems. Set up alerts for unusual patterns. Store logs with appropriate retention for audit compliance.

AI regression testing verifies that model updates do not degrade performance on previously working cases. Unlike software regression testing with deterministic pass/fail, AI regression testing deals with probabilistic outputs and requires statistical comparison. Key differences include tolerance thresholds for quality changes, the need for representative test distributions, and the possibility of acceptable trade-offs between metrics.

A/B testing compares two model versions with real users by randomly assigning traffic. Design: define a clear hypothesis and primary metric, calculate required sample size for statistical power, randomise at the user level to avoid cross-contamination, run for sufficient duration to capture temporal patterns, and use proper statistical tests (not just point estimates) for decision making.

Development evaluation uses curated test sets to make go/no-go decisions before deployment, focusing on model quality and safety. Production monitoring tracks live performance metrics, user feedback, latency, error rates, and data drift continuously. Production monitoring catches issues that test sets miss, like distribution shifts, real-world edge cases, and gradual performance degradation over time.

A feature store is a centralised repository for storing, managing, and serving features for ML models. It ensures consistency between training and serving features, enables feature reuse across teams, handles feature versioning, and provides both batch and real-time feature serving. Popular options include Feast, Tecton, and Hopsworks. It prevents training-serving skew.

CI includes automated testing of data pipeline code, feature engineering logic, and model training scripts. CD automates model deployment through staging and production environments. Include data validation checks, model quality gates (minimum accuracy thresholds), canary deployments, and automated rollback. Use tools like GitHub Actions, Jenkins, or Kubeflow Pipelines. Version both code and data artifacts.

Model versioning tracks different iterations of models including weights, hyperparameters, training data, and code. Use a model registry (MLflow, Weights & Biases, or cloud-native options) to store and manage versions. Track lineage (which data and code produced each model). Implement promotion stages (development, staging, production). Enable rollback to previous versions when issues arise.

Data drift occurs when the statistical properties of production data change from training data distribution. Detect it using statistical tests (KS test, PSI) on feature distributions, monitoring prediction confidence distributions, and tracking feature statistics over time. Handle it by triggering retraining pipelines, alerting data teams, and implementing adaptive models. Monitor continuously, not just at deployment.

Use a traffic splitting layer to route requests to different model versions. Randomise at the user level for consistency. Define clear success metrics and minimum sample sizes before starting. Run for sufficient duration to capture weekly patterns. Use proper statistical tests and correct for multiple comparisons. Implement guardrail metrics that trigger automatic rollback if violated.

Training-serving skew occurs when the feature computation during training differs from serving, leading to degraded model performance. Causes include different code paths, different libraries, or timing differences. Prevent it by using a feature store for consistent feature computation, sharing preprocessing code between training and serving, and implementing data validation checks.

Monitor prediction distributions for drift. Track model performance metrics against ground truth when available. Monitor input data quality and feature distributions. Set up alerts for anomalous prediction patterns. Track infrastructure metrics (latency, throughput, error rates, GPU utilisation). Implement dashboards for real-time visibility. Use tools like Evidently, WhyLabs, or Arize for ML-specific monitoring.

Level 0: Manual, ad-hoc ML with no automation. Level 1: ML pipeline automation with continuous training but manual deployment. Level 2: CI/CD for both ML pipelines and model deployment with automated testing. Level 3: Full automation with continuous monitoring, automated retraining triggered by drift detection, and self-healing systems. Most Indian companies are between levels 0 and 1.

Maintain versioned model artifacts in a model registry with deployment history. Implement blue-green or canary deployment strategies. Monitor canary metrics with automatic rollback triggers. Keep the previous model version warm for instant switching. Store rollback procedures as runbooks. Test rollback procedures regularly. Ensure the data pipeline supports the rolled-back model's feature requirements.

Kubeflow is an open-source ML platform built on Kubernetes that provides components for the full ML lifecycle. It includes Kubeflow Pipelines for workflow orchestration, KFServing for model serving, Katib for hyperparameter tuning, and notebook servers for development. It enables scalable, reproducible ML workflows on any Kubernetes cluster, popular in Indian enterprises using on-premise infrastructure.

Pin all library versions in requirements files. Use Docker containers for complete environment reproducibility. Store training data versions with DVC or similar tools. Track random seeds and hardware specifications. Record exact Git commits for training code. Use deterministic training where possible. Document CUDA and driver versions for GPU training. Automate environment setup in CI/CD.

Canary deployment routes a small percentage of production traffic (e.g., 5%) to the new model while the majority uses the current model. If the canary model performs well on defined metrics, traffic is gradually increased. If it degrades, traffic is rolled back automatically. This minimises risk by limiting the blast radius of a bad model deployment.

Define data schemas with expected types, ranges, and constraints using tools like Great Expectations or TFX Data Validation. Check for missing values, outliers, and distribution changes. Validate referential integrity and business rules. Run validation at data ingestion, after feature engineering, and before training. Fail the pipeline early if validation fails. Generate data quality reports.

DVC tracks large files, datasets, and ML model artifacts alongside Git without storing them in the Git repository itself. It stores metadata in Git while the actual data lives in remote storage (S3, GCS, Azure Blob). DVC enables reproducible ML experiments by versioning data alongside code, supports pipeline definition, and enables data sharing across teams.

Model compression reduces model size and inference cost for production deployment. Techniques include quantisation (reducing precision from FP32 to INT8/INT4), pruning (removing unimportant weights or neurons), knowledge distillation (training a smaller model), and architecture search (finding efficient architectures). Choose based on acceptable quality loss and target deployment hardware.

Never hardcode secrets in code or configuration files. Use secret management services (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault). Inject secrets as environment variables at runtime. Use IAM roles and service accounts instead of API keys where possible. Rotate credentials regularly. Audit secret access. Scan repositories for accidentally committed secrets using tools like git-secrets.

Tag all ML resources for cost attribution. Use spot/preemptible instances for training (60-90% savings). Right-size GPU instances based on actual utilisation. Implement auto-scaling for serving. Use model optimisation (quantisation, distillation) to reduce inference costs. Schedule batch jobs during off-peak hours. Set up budget alerts and cost anomaly detection. Review costs weekly.

A model registry is a centralised catalog of trained models with their metadata. Store: model artifacts (weights, configs), training metrics, data lineage (dataset version, features used), code version (git commit), hyperparameters, evaluation results, deployment status, owner/team, creation date, and model card. Popular registries include MLflow Model Registry, Vertex AI Model Registry, and SageMaker Model Registry.

Implement structured logging at each pipeline stage. Use monitoring tools (Datadog, Grafana, CloudWatch) for pipeline health dashboards. Set up alerts for failures, SLA breaches, and quality degradation. Implement retry logic with exponential backoff for transient failures. Create runbooks for common failure scenarios. Use dead letter queues for failed data records. Post-mortem significant incidents.

LLMOps adapts MLOps practices for LLM-based applications. Key differences include prompt management and versioning instead of feature engineering, evaluation through LLM-as-judge and human preference, fine-tuning with PEFT instead of full training, managing API costs and rate limits, monitoring for hallucination and safety issues, and handling much larger model artifacts and compute requirements.

Define scaling metrics: GPU utilisation, request queue depth, latency percentiles, or custom business metrics. Set target values and scaling thresholds. Use Kubernetes HPA (Horizontal Pod Autoscaler) or cloud auto-scaling groups. Implement warm-up time for new instances (model loading takes time). Set minimum replicas for baseline traffic. Use predictive scaling for known traffic patterns.

ONNX (Open Neural Network Exchange) is an open format for representing ML models, enabling interoperability between frameworks. You can train in PyTorch and deploy with ONNX Runtime, TensorRT, or OpenVINO. ONNX Runtime provides optimised inference across CPU, GPU, and edge devices. This decouples training framework choice from deployment infrastructure, enabling hardware-specific optimisations.

Kubernetes orchestrates containerised applications across clusters of machines. For ML, it provides GPU scheduling and sharing, auto-scaling of inference services, resource isolation for multiple teams, rolling deployments with zero downtime, self-healing when containers fail, and infrastructure-as-code for reproducibility. It is the standard platform for production ML serving.

Use DCGM (Data Center GPU Manager) for GPU metrics. Monitor GPU utilisation, memory usage, temperature, power consumption, and ECC errors. Integrate with Prometheus for collection and Grafana for visualisation. Set alerts for low utilisation (wasted resources), high temperature (throttling risk), and ECC errors (hardware failure). Track GPU time per team for chargeback. Monitor driver and CUDA version consistency.

Triton is an open-source inference serving platform that supports multiple frameworks (PyTorch, TensorFlow, ONNX, TensorRT) simultaneously. Key features include dynamic batching, model versioning, model ensembles, GPU scheduling across multiple models, metrics reporting, and HTTP/gRPC APIs. It is production-ready and widely used in enterprise deployments for serving multiple models efficiently.

Use transfer learning with a pre-trained CNN like EfficientNet fine-tuned on labelled crop disease images. The model should handle diverse Indian crops and work offline on low-end devices via model quantisation. Include data augmentation for varying lighting and camera quality. Deploy as a lightweight mobile app with results in local languages.

Use a combination of rule-based filters for known fraud patterns and ML models (gradient boosting or deep learning) for anomaly detection. Feature engineer from transaction metadata, user behavior, device info, and network graphs. Handle extreme class imbalance with SMOTE or focal loss. Deploy with real-time inference under 100ms latency and continuous model retraining.

Use Apache Kafka or Redis Streams for message ingestion, asyncio or Faust for stream processing, and a FastAPI/gRPC endpoint for inference. Implement windowed feature computation with a sliding buffer. Use connection pooling for database lookups and batch inference requests when possible. Monitor latency and throughput with Prometheus metrics.

Route a percentage of traffic to each prompt variant. Define clear metrics (accuracy, user satisfaction, latency, cost). Ensure sufficient sample size for statistical significance. Log all inputs, outputs, and metadata. Use automated evaluation with LLM judges plus manual review of edge cases. Implement gradual rollout with rollback capability. Monitor for regression over time.

Cache frequent queries and their results. Use approximate nearest neighbour indices for faster search. Implement query embedding caching. Batch vector lookups. Use streaming for LLM generation. Pre-compute and store embeddings for static content. Optimise chunk sizes to reduce the number of retrievals. Consider using smaller, faster models for retrieval stages.

Implement input validation before tool execution. Set maximum iteration and cost limits. Use content filtering on outputs. Require human approval for high-impact actions (sending emails, modifying data). Implement allowlists for permitted tool actions. Add circuit breakers for repeated failures. Log all actions for audit trails. Monitor for anomalous behaviour patterns.

Equip the agent with tools for monitoring (Grafana/CloudWatch queries), deployment (kubectl, CI/CD triggers), log analysis, and alert management. Implement strict approval workflows for production changes. Start with read-only tools and gradually add write capabilities with human approval. Handle Indian cloud regions (Mumbai, Hyderabad). Include incident response runbooks as context. Monitor agent actions carefully during the initial rollout period.

Use on-premise or private cloud GPU infrastructure. Apply differential privacy during training to prevent memorisation. Anonymise or pseudonymise PII in training data. Use federated fine-tuning when data cannot leave its source. Implement access controls on model artifacts. Audit the model for data leakage using membership inference tests. Comply with India's DPDP Act requirements.

Integrate eval scripts into the CI pipeline triggered by model or prompt changes. Store test sets and expected outputs in version control. Run automated metrics computation and LLM-as-judge evaluations. Set quality gates with minimum threshold scores. Generate evaluation reports comparing against baselines. Alert on regressions. Use tools like promptfoo, Braintrust, or custom evaluation frameworks.

Use a model serving framework (TorchServe, Triton, or TF Serving) behind a load balancer. Deploy in Indian cloud regions (AWS Mumbai, Azure Central India) for low latency. Implement auto-scaling based on traffic patterns (handle sale events like Diwali/Republic Day). Use model caching and batching for efficiency. Set up health checks, monitoring, and automatic failover. Consider edge deployment for latency-critical features.

Define triggers: scheduled (daily/weekly), data drift detection, or performance degradation below threshold. Automate data fetching, preprocessing, training, and evaluation. Implement quality gates that compare new model against current production model. Use automated deployment with canary rollout for approved models. Alert on failures and unusual metrics. Track retraining history for audit.

Route production traffic to both the current and new model simultaneously. The current model serves responses to users while the new model's predictions are logged but not shown. Compare predictions offline to detect discrepancies and evaluate the new model on real traffic. This is risk-free testing on live data. Implement using a proxy layer or traffic mirroring at the load balancer.

Start with managed services to move fast: GitHub for code, DVC for data versioning, MLflow for experiment tracking, AWS SageMaker or GCP Vertex AI for training and serving. Implement CI/CD with GitHub Actions. Use Docker and Kubernetes for serving. Set up monitoring with Prometheus and Grafana. Automate data validation and model evaluation. Plan for compliance with Indian data regulations. Scale infrastructure as the team grows.

Use GPU-optimised inference servers (vLLM, TGI, or Triton) with continuous batching. Quantise the model to INT8/INT4 for speed. Implement KV cache efficiently. Deploy in regions close to users (Indian cloud regions). Use load balancing with health checks. Implement request queuing and timeout handling. Pre-warm model caches. Monitor P99 latency and auto-scale based on queue depth.

Use a streaming platform (Kafka, Kinesis) for event ingestion. Process events with a stream processor (Flink, Spark Streaming) for feature computation. Store computed features in a low-latency store (Redis, DynamoDB) for serving. Implement exactly-once processing semantics. Handle late-arriving data with watermarks. Monitor pipeline lag and throughput. Ensure consistency with batch-computed features.

Replicate model artifacts and configurations across regions. Use multi-region deployment with failover routing. Implement automated health checks and failover triggers. Back up model registries and feature stores. Document recovery procedures and test them regularly. Define RTO (Recovery Time Objective) and RPO (Recovery Point Objective). Keep model serving stateless for easy recovery.

Deploy Triton or a custom serving layer on Kubernetes with GPU node pools in Indian cloud regions. Implement model routing based on request type. Use shared GPU scheduling to maximise utilisation across models. Implement a model registry for version management. Set up centralised monitoring and cost attribution per team. Handle compliance with Indian data regulations. Implement rate limiting and authentication per consumer.