MLOps Engineer

Use virtual environments (venv, conda, or poetry) to isolate project dependencies. Pin exact versions in requirements.txt or pyproject.toml for reproducibility. Use conda for complex native dependencies like CUDA. Docker containers ensure full environment reproducibility. Tools like pip-tools or poetry provide dependency resolution and lock files.

MCP is an open protocol that standardises how AI applications connect to external data sources and tools. It solves the M x N integration problem where each AI app would need custom integrations with each tool. With MCP, tools implement the protocol once and any MCP-compatible AI application can use them, similar to how USB standardised device connections.

Evals are systematic methods for measuring AI system performance across quality, safety, and reliability dimensions. They are critical because they prevent deploying underperforming or harmful models, enable data-driven decision making between model versions, catch regressions early, and build stakeholder confidence. Without evals, AI development is essentially flying blind.

Offline evaluation tests models on static datasets before deployment using metrics like accuracy, F1, and perplexity. Online evaluation measures model performance in production with real users through A/B testing, user satisfaction, and business metrics. Offline catches obvious issues early; online captures real-world performance that test sets cannot fully represent. Both are essential.

MLOps (Machine Learning Operations) applies DevOps principles to ML systems, covering the full lifecycle from data preparation through model deployment and monitoring. It is important because most ML models fail to reach production without proper engineering practices. MLOps ensures reproducibility, reliability, scalability, and continuous improvement of ML systems in production environments.

An ML pipeline includes data ingestion and validation, feature engineering and storage, model training and hyperparameter tuning, model evaluation and validation, model registry and versioning, deployment (serving infrastructure), monitoring and alerting, and feedback loops for retraining. Each component should be automated, reproducible, and version-controlled.

Batch inference processes large volumes of data periodically (hourly, daily) and stores results for later use. Real-time inference processes individual requests with low latency (milliseconds). Batch is simpler and cheaper for recommendations or scoring; real-time is needed for fraud detection, search ranking, and conversational AI. Many systems use both: batch for pre-computation and real-time for personalisation.

Experiment tracking records hyperparameters, metrics, code versions, data versions, and artifacts for each training run. It enables comparison across experiments and reproducibility. Popular tools include MLflow (open-source, self-hosted), Weights & Biases (cloud-based with rich visualisation), Neptune, and Comet. Choose based on team size, budget, and integration requirements.

Model deployment is the process of making a model available in a production environment, including infrastructure setup, configuration, and release management. Model serving is the runtime system that handles inference requests, including request batching, model loading, scaling, and response formatting. Deployment is a one-time event; serving is the ongoing operational system.

Docker containers package the model, code, dependencies, and runtime into a portable, reproducible unit. This eliminates 'works on my machine' issues, ensures consistent behaviour across environments, simplifies deployment, and enables scaling via orchestrators like Kubernetes. Use multi-stage builds to minimise image size and separate training from serving containers.

GPUs (NVIDIA A100, H100) are the dominant hardware for training and inference due to their massive parallelism. TPUs (Google's custom chips) are optimised for tensor operations. CPUs handle preprocessing and lightweight inference. Specialised accelerators like AWS Inferentia target inference cost-efficiency. FPGAs offer programmable acceleration for specific workloads.

CUDA is NVIDIA's parallel computing platform that enables general-purpose GPU programming. It provides APIs for launching parallel computations on GPU cores. Deep learning frameworks (PyTorch, TensorFlow) use CUDA internally for accelerated tensor operations. CUDA's ecosystem including cuBLAS, cuDNN, and NCCL is the primary reason NVIDIA dominates the AI hardware market.

Vertical scaling adds more resources (bigger GPU, more memory) to a single instance, limited by hardware maximums. Horizontal scaling adds more instances behind a load balancer, limited by coordination complexity. For ML serving, use vertical scaling for model size constraints (larger model needs bigger GPU) and horizontal scaling for throughput. Most production systems combine both approaches.

Evaluate on task-specific benchmarks with your actual data, not just public leaderboards. Compare latency, throughput, cost per token, context window size, and output quality. Consider factors like API reliability, data privacy policies, and hosting options (cloud vs self-hosted). Run A/B tests and collect human evaluation scores on representative queries.

Test each transformation function with known inputs and expected outputs. Test edge cases like empty DataFrames, missing values, and unexpected types. Use pytest fixtures for reusable test data. Test the full pipeline end-to-end with a small sample. Verify output shapes, dtypes, and value ranges. Use hypothesis library for property-based testing of transformations.

Pickle serialises Python objects to byte streams for storage and loading. It is commonly used to save scikit-learn models and data. However, pickle can execute arbitrary code during deserialisation, making it a security risk for untrusted sources. Use safer alternatives like ONNX, safetensors, or joblib with known-source files. Never unpickle data from untrusted origins.

Use FastAPI or Flask to create endpoints. Load the model at startup (not per request). Define Pydantic models for request/response validation. Implement async handlers for IO-bound operations. Add health checks, logging, and error handling. Use gunicorn/uvicorn with multiple workers for production. Containerise with Docker and add rate limiting.

Prompt templates are parameterised prompts with placeholders filled at runtime (e.g., 'Summarise this {document_type} in {language}'). In production, manage them as versioned configuration with template engines like Jinja2. Store in a prompt registry, track performance per version, and implement rollback capability. LangChain and similar frameworks provide prompt template abstractions.

Remove redundant instructions and examples. Use concise, precise language. Cache common prompt prefixes where the API supports it. Replace verbose few-shot examples with clear instructions when possible. Use shorter output formats (abbreviations, codes instead of full text). Batch similar requests. Evaluate whether a smaller, cheaper model suffices for the task.

Implement incremental indexing that detects changed documents via checksums or timestamps. Use metadata to track document versions and update timestamps. For critical applications, maintain version history and allow querying against specific versions. Implement a document deletion pipeline that removes stale vectors. Schedule periodic full re-indexing for consistency.

Monitor retrieval latency, embedding computation time, and end-to-end response time. Track retrieval hit rate and relevance scores. Measure answer faithfulness and hallucination rate. Monitor index size growth and query volume. Track user feedback (thumbs up/down, corrections). Alert on unusual query patterns, error rates, or degradation in response quality.

MCP implements security through transport-level encryption (TLS for SSE), OAuth 2.0 for authentication with remote servers, and a human-in-the-loop approval model where the host application can require user confirmation before tool execution. Servers should validate inputs, implement rate limiting, and follow least-privilege principles. Sensitive operations should always require explicit user consent.

Implement per-client rate limiting using token buckets or sliding window counters. Return appropriate error codes (429) with retry-after headers. Track usage per authenticated user. Set tool-specific rate limits based on the cost of the underlying operation. Communicate limits through tool descriptions. Cache frequent requests. Consider implementing priority queues for different user tiers.

Use the protocol version field in the initialise handshake. Add new tools without removing old ones. Deprecate tools by updating their descriptions before removal. Make new parameters optional with defaults. Use semantic versioning for the server itself. Test new versions against existing clients. Document breaking changes clearly. Support multiple versions simultaneously during transitions.

Use MCP's built-in logging capability to send structured log messages to the client. Log all tool invocations with parameters, execution time, and outcomes. Implement correlation IDs to trace requests across the system. Export metrics (latency, error rate, usage) to monitoring systems. Set up alerts for unusual patterns. Store logs with appropriate retention for audit compliance.

Multiple servers follow the Unix philosophy of doing one thing well, enabling independent development, deployment, and scaling. A monolithic server is simpler to deploy but harder to maintain. Compose multiple servers when tools have different security requirements, update frequencies, or team ownership. Use a monolithic server for tightly coupled tools that share state.

MCP adds a layer of abstraction and latency that may be unnecessary for simple, single-tool integrations. It requires both client and server support. The ecosystem is still maturing with limited SDKs in some languages. For high-throughput, low-latency requirements, direct API integration may be more appropriate. If you only integrate with one or two tools, the protocol overhead may not be justified.

Agent loops consume many LLM tokens through repeated reasoning, tool calls, and observations. Costs multiply with retries, long context, and multi-agent coordination. Optimise by using cheaper models for simple tool selection, caching frequent operations, limiting iteration counts, compressing context, and batching similar tasks. Monitor cost per completed task and set budgets per workflow.

For a 7B model: QLoRA needs ~16GB VRAM (single consumer GPU), full fine-tuning needs ~112GB (2x A100). For 13B: QLoRA needs ~24GB, full needs ~208GB. For 70B: QLoRA needs ~48GB (single A100), full needs ~1TB+ (multi-node). Training time scales roughly linearly with data size. Use gradient checkpointing and DeepSpeed ZeRO to reduce memory at the cost of speed.

AI regression testing verifies that model updates do not degrade performance on previously working cases. Unlike software regression testing with deterministic pass/fail, AI regression testing deals with probabilistic outputs and requires statistical comparison. Key differences include tolerance thresholds for quality changes, the need for representative test distributions, and the possibility of acceptable trade-offs between metrics.

Measure P50, P95, and P99 latency at each pipeline stage (preprocessing, model inference, postprocessing). Profile with tools like NVIDIA Nsight for GPU utilisation. Optimise by reducing model size (quantisation, distillation), batching requests, using faster hardware, implementing caching, and optimising tokenisation. Set latency SLAs based on user experience requirements.

Development evaluation uses curated test sets to make go/no-go decisions before deployment, focusing on model quality and safety. Production monitoring tracks live performance metrics, user feedback, latency, error rates, and data drift continuously. Production monitoring catches issues that test sets miss, like distribution shifts, real-world edge cases, and gradual performance degradation over time.

A feature store is a centralised repository for storing, managing, and serving features for ML models. It ensures consistency between training and serving features, enables feature reuse across teams, handles feature versioning, and provides both batch and real-time feature serving. Popular options include Feast, Tecton, and Hopsworks. It prevents training-serving skew.

CI includes automated testing of data pipeline code, feature engineering logic, and model training scripts. CD automates model deployment through staging and production environments. Include data validation checks, model quality gates (minimum accuracy thresholds), canary deployments, and automated rollback. Use tools like GitHub Actions, Jenkins, or Kubeflow Pipelines. Version both code and data artifacts.

Model versioning tracks different iterations of models including weights, hyperparameters, training data, and code. Use a model registry (MLflow, Weights & Biases, or cloud-native options) to store and manage versions. Track lineage (which data and code produced each model). Implement promotion stages (development, staging, production). Enable rollback to previous versions when issues arise.

Data drift occurs when the statistical properties of production data change from training data distribution. Detect it using statistical tests (KS test, PSI) on feature distributions, monitoring prediction confidence distributions, and tracking feature statistics over time. Handle it by triggering retraining pipelines, alerting data teams, and implementing adaptive models. Monitor continuously, not just at deployment.

Use a traffic splitting layer to route requests to different model versions. Randomise at the user level for consistency. Define clear success metrics and minimum sample sizes before starting. Run for sufficient duration to capture weekly patterns. Use proper statistical tests and correct for multiple comparisons. Implement guardrail metrics that trigger automatic rollback if violated.

Training-serving skew occurs when the feature computation during training differs from serving, leading to degraded model performance. Causes include different code paths, different libraries, or timing differences. Prevent it by using a feature store for consistent feature computation, sharing preprocessing code between training and serving, and implementing data validation checks.

Monitor prediction distributions for drift. Track model performance metrics against ground truth when available. Monitor input data quality and feature distributions. Set up alerts for anomalous prediction patterns. Track infrastructure metrics (latency, throughput, error rates, GPU utilisation). Implement dashboards for real-time visibility. Use tools like Evidently, WhyLabs, or Arize for ML-specific monitoring.

Level 0: Manual, ad-hoc ML with no automation. Level 1: ML pipeline automation with continuous training but manual deployment. Level 2: CI/CD for both ML pipelines and model deployment with automated testing. Level 3: Full automation with continuous monitoring, automated retraining triggered by drift detection, and self-healing systems. Most Indian companies are between levels 0 and 1.

Maintain versioned model artifacts in a model registry with deployment history. Implement blue-green or canary deployment strategies. Monitor canary metrics with automatic rollback triggers. Keep the previous model version warm for instant switching. Store rollback procedures as runbooks. Test rollback procedures regularly. Ensure the data pipeline supports the rolled-back model's feature requirements.

Kubeflow is an open-source ML platform built on Kubernetes that provides components for the full ML lifecycle. It includes Kubeflow Pipelines for workflow orchestration, KFServing for model serving, Katib for hyperparameter tuning, and notebook servers for development. It enables scalable, reproducible ML workflows on any Kubernetes cluster, popular in Indian enterprises using on-premise infrastructure.

Pin all library versions in requirements files. Use Docker containers for complete environment reproducibility. Store training data versions with DVC or similar tools. Track random seeds and hardware specifications. Record exact Git commits for training code. Use deterministic training where possible. Document CUDA and driver versions for GPU training. Automate environment setup in CI/CD.

Canary deployment routes a small percentage of production traffic (e.g., 5%) to the new model while the majority uses the current model. If the canary model performs well on defined metrics, traffic is gradually increased. If it degrades, traffic is rolled back automatically. This minimises risk by limiting the blast radius of a bad model deployment.

Define data schemas with expected types, ranges, and constraints using tools like Great Expectations or TFX Data Validation. Check for missing values, outliers, and distribution changes. Validate referential integrity and business rules. Run validation at data ingestion, after feature engineering, and before training. Fail the pipeline early if validation fails. Generate data quality reports.

DVC tracks large files, datasets, and ML model artifacts alongside Git without storing them in the Git repository itself. It stores metadata in Git while the actual data lives in remote storage (S3, GCS, Azure Blob). DVC enables reproducible ML experiments by versioning data alongside code, supports pipeline definition, and enables data sharing across teams.

Model compression reduces model size and inference cost for production deployment. Techniques include quantisation (reducing precision from FP32 to INT8/INT4), pruning (removing unimportant weights or neurons), knowledge distillation (training a smaller model), and architecture search (finding efficient architectures). Choose based on acceptable quality loss and target deployment hardware.

Never hardcode secrets in code or configuration files. Use secret management services (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault). Inject secrets as environment variables at runtime. Use IAM roles and service accounts instead of API keys where possible. Rotate credentials regularly. Audit secret access. Scan repositories for accidentally committed secrets using tools like git-secrets.

Tag all ML resources for cost attribution. Use spot/preemptible instances for training (60-90% savings). Right-size GPU instances based on actual utilisation. Implement auto-scaling for serving. Use model optimisation (quantisation, distillation) to reduce inference costs. Schedule batch jobs during off-peak hours. Set up budget alerts and cost anomaly detection. Review costs weekly.

A model registry is a centralised catalog of trained models with their metadata. Store: model artifacts (weights, configs), training metrics, data lineage (dataset version, features used), code version (git commit), hyperparameters, evaluation results, deployment status, owner/team, creation date, and model card. Popular registries include MLflow Model Registry, Vertex AI Model Registry, and SageMaker Model Registry.

Implement structured logging at each pipeline stage. Use monitoring tools (Datadog, Grafana, CloudWatch) for pipeline health dashboards. Set up alerts for failures, SLA breaches, and quality degradation. Implement retry logic with exponential backoff for transient failures. Create runbooks for common failure scenarios. Use dead letter queues for failed data records. Post-mortem significant incidents.

LLMOps adapts MLOps practices for LLM-based applications. Key differences include prompt management and versioning instead of feature engineering, evaluation through LLM-as-judge and human preference, fine-tuning with PEFT instead of full training, managing API costs and rate limits, monitoring for hallucination and safety issues, and handling much larger model artifacts and compute requirements.

The H100 offers approximately 3x the performance of A100 for transformer training thanks to the Transformer Engine with FP8 support, higher memory bandwidth (3.35 TB/s vs 2 TB/s), and more CUDA cores. H100 has 80GB HBM3 memory vs A100's 80GB HBM2e. H100 also introduces NVLink 4.0 for faster multi-GPU communication. H100 is preferred for large model training and inference.

Cloud offers flexibility, scalability, and no upfront capital. On-premise provides lower long-term costs for sustained workloads and data sovereignty. Cloud is better for experimentation, burst capacity, and teams without infrastructure expertise. On-premise makes sense for consistent high utilisation (>70%), sensitive data requirements (Indian data localisation), or large-scale training where reserved instances cost more than owned hardware.

Profile GPU utilisation with nvidia-smi and PyTorch profiler. Increase batch size to fill GPU memory. Use mixed precision training (FP16/BF16) for higher throughput. Optimise data loading with multiple workers and prefetching. Enable gradient checkpointing to trade compute for memory. Use DeepSpeed or FSDP for multi-GPU efficiency. Reduce CPU-GPU data transfer bottlenecks.

Define scaling metrics: GPU utilisation, request queue depth, latency percentiles, or custom business metrics. Set target values and scaling thresholds. Use Kubernetes HPA (Horizontal Pod Autoscaler) or cloud auto-scaling groups. Implement warm-up time for new instances (model loading takes time). Set minimum replicas for baseline traffic. Use predictive scaling for known traffic patterns.

Evaluate: query performance (latency at scale), scalability (data size and QPS), index types supported (HNSW, IVF), filtering capabilities (metadata filters), managed vs self-hosted, cost model, backup and replication, cloud region availability (Indian regions), API compatibility with your stack, and community/support quality. Popular choices include Pinecone, Weaviate, Qdrant, Milvus, and pgvector.

ONNX (Open Neural Network Exchange) is an open format for representing ML models, enabling interoperability between frameworks. You can train in PyTorch and deploy with ONNX Runtime, TensorRT, or OpenVINO. ONNX Runtime provides optimised inference across CPU, GPU, and edge devices. This decouples training framework choice from deployment infrastructure, enabling hardware-specific optimisations.

Kubernetes orchestrates containerised applications across clusters of machines. For ML, it provides GPU scheduling and sharing, auto-scaling of inference services, resource isolation for multiple teams, rolling deployments with zero downtime, self-healing when containers fail, and infrastructure-as-code for reproducibility. It is the standard platform for production ML serving.

Edge deployment runs AI models on devices near the data source (phones, IoT devices, local servers) rather than in the cloud. It is appropriate when low latency is critical, network connectivity is unreliable (rural India), data privacy prevents cloud transmission, or cloud costs are prohibitive at scale. Requires model compression (quantisation, pruning) for device constraints.

Use object storage (S3, GCS, Azure Blob) as the foundation. Implement a medallion architecture: bronze (raw data), silver (cleaned and validated), gold (feature-ready). Use Delta Lake or Iceberg for ACID transactions and time travel. Catalog data with Apache Hive or Unity Catalog. Implement access controls and data lineage tracking. Optimise with columnar formats (Parquet) and partitioning.

Cache at multiple levels: model weights in GPU memory (avoid reloading), KV cache for conversation history, query-result cache for frequent queries (Redis/Memcached), and embedding cache for common inputs. Implement cache invalidation based on model version changes. Use LRU eviction for memory-bounded caches. Monitor cache hit rates and adjust sizing accordingly.

Use DCGM (Data Center GPU Manager) for GPU metrics. Monitor GPU utilisation, memory usage, temperature, power consumption, and ECC errors. Integrate with Prometheus for collection and Grafana for visualisation. Set alerts for low utilisation (wasted resources), high temperature (throttling risk), and ECC errors (hardware failure). Track GPU time per team for chargeback. Monitor driver and CUDA version consistency.

Triton is an open-source inference serving platform that supports multiple frameworks (PyTorch, TensorFlow, ONNX, TensorRT) simultaneously. Key features include dynamic batching, model versioning, model ensembles, GPU scheduling across multiple models, metrics reporting, and HTTP/gRPC APIs. It is production-ready and widely used in enterprise deployments for serving multiple models efficiently.

Serverless inference (AWS Lambda, Google Cloud Functions, Azure Functions) auto-scales to zero when idle and scales up on demand. It eliminates infrastructure management and charges only for actual usage. Appropriate for low-traffic or bursty ML endpoints with small models. Not suitable for large models (memory limits), GPU workloads, or latency-sensitive applications due to cold start times.

KV cache stores the key and value tensors computed for previously processed tokens so they do not need to be recomputed at each generation step. This reduces inference time from O(n^2) to O(n) per token. However, it requires significant GPU memory, especially for long sequences and large batch sizes.

In FP16, each parameter takes 2 bytes, so 70B parameters need approximately 140 GB just for model weights. Add KV cache memory which depends on context length and batch size. A rough estimate for serving would be 160-200 GB, requiring multiple A100 (80GB) or H100 GPUs with tensor parallelism. Quantisation to INT4 can reduce this to around 40 GB.

Quantisation reduces model weights from higher precision (FP32/FP16) to lower precision (INT8/INT4), dramatically reducing memory usage and improving inference speed. Trade-offs include slight quality degradation, especially for complex reasoning tasks. Techniques like GPTQ, AWQ, and GGML enable 4-bit quantisation with minimal quality loss for many applications.

Use Apache Kafka or Redis Streams for message ingestion, asyncio or Faust for stream processing, and a FastAPI/gRPC endpoint for inference. Implement windowed feature computation with a sliding buffer. Use connection pooling for database lookups and batch inference requests when possible. Monitor latency and throughput with Prometheus metrics.

Use torch.nn.parallel.DistributedDataParallel (DDP) for data parallelism. Initialise process groups with torch.distributed.init_process_group. Wrap the model in DDP and use DistributedSampler for the DataLoader. Launch with torchrun or torch.distributed.launch. For model parallelism across GPUs, use pipeline parallelism or tensor parallelism with libraries like DeepSpeed or FSDP.

Route a percentage of traffic to each prompt variant. Define clear metrics (accuracy, user satisfaction, latency, cost). Ensure sufficient sample size for statistical significance. Log all inputs, outputs, and metadata. Use automated evaluation with LLM judges plus manual review of edge cases. Implement gradual rollout with rollback capability. Monitor for regression over time.

Store access control metadata (tenant, department, role) with each chunk. Apply metadata filters at query time to enforce permissions. Integrate with the enterprise's existing IAM (like Active Directory). Ensure vector database supports metadata filtering efficiently. Implement audit logging for compliance. Handle Indian data localisation requirements by keeping sensitive data in India-based storage.

Cache frequent queries and their results. Use approximate nearest neighbour indices for faster search. Implement query embedding caching. Batch vector lookups. Use streaming for LLM generation. Pre-compute and store embeddings for static content. Optimise chunk sizes to reduce the number of retrievals. Consider using smaller, faster models for retrieval stages.

Create separate MCP servers for each domain: HR (employee data, leave management), Finance (invoicing, expense reports), CRM (customer data), and IT (ticket management). Implement a gateway server for authentication and routing. Use SSE transport for shared remote servers. Standardise tool naming and error handling across servers. Document tools for both LLMs and developers.

Implement input validation before tool execution. Set maximum iteration and cost limits. Use content filtering on outputs. Require human approval for high-impact actions (sending emails, modifying data). Implement allowlists for permitted tool actions. Add circuit breakers for repeated failures. Log all actions for audit trails. Monitor for anomalous behaviour patterns.

Use a persistent state store (database or Redis) for workflow state. Implement checkpointing at each step so workflows can resume after failures. Serialise state in a structured format with version tracking. Handle concurrent access with locking mechanisms. Implement state cleanup for abandoned workflows. Use event sourcing for full auditability of state changes.

Agents face prompt injection through tool outputs, unintended tool execution that modifies systems, data exfiltration through crafted tool calls, privilege escalation by chaining permissions, and denial-of-service through infinite loops. Mitigate with strict input validation, least-privilege tool access, output sanitisation, action approval workflows, and comprehensive audit logging.

Equip the agent with tools for monitoring (Grafana/CloudWatch queries), deployment (kubectl, CI/CD triggers), log analysis, and alert management. Implement strict approval workflows for production changes. Start with read-only tools and gradually add write capabilities with human approval. Handle Indian cloud regions (Mumbai, Hyderabad). Include incident response runbooks as context. Monitor agent actions carefully during the initial rollout period.

Use on-premise or private cloud GPU infrastructure. Apply differential privacy during training to prevent memorisation. Anonymise or pseudonymise PII in training data. Use federated fine-tuning when data cannot leave its source. Implement access controls on model artifacts. Audit the model for data leakage using membership inference tests. Comply with India's DPDP Act requirements.

Integrate eval scripts into the CI pipeline triggered by model or prompt changes. Store test sets and expected outputs in version control. Run automated metrics computation and LLM-as-judge evaluations. Set quality gates with minimum threshold scores. Generate evaluation reports comparing against baselines. Alert on regressions. Use tools like promptfoo, Braintrust, or custom evaluation frameworks.

Use a model serving framework (TorchServe, Triton, or TF Serving) behind a load balancer. Deploy in Indian cloud regions (AWS Mumbai, Azure Central India) for low latency. Implement auto-scaling based on traffic patterns (handle sale events like Diwali/Republic Day). Use model caching and batching for efficiency. Set up health checks, monitoring, and automatic failover. Consider edge deployment for latency-critical features.

Define triggers: scheduled (daily/weekly), data drift detection, or performance degradation below threshold. Automate data fetching, preprocessing, training, and evaluation. Implement quality gates that compare new model against current production model. Use automated deployment with canary rollout for approved models. Alert on failures and unusual metrics. Track retraining history for audit.

Route production traffic to both the current and new model simultaneously. The current model serves responses to users while the new model's predictions are logged but not shown. Compare predictions offline to detect discrepancies and evaluate the new model on real traffic. This is risk-free testing on live data. Implement using a proxy layer or traffic mirroring at the load balancer.

Start with managed services to move fast: GitHub for code, DVC for data versioning, MLflow for experiment tracking, AWS SageMaker or GCP Vertex AI for training and serving. Implement CI/CD with GitHub Actions. Use Docker and Kubernetes for serving. Set up monitoring with Prometheus and Grafana. Automate data validation and model evaluation. Plan for compliance with Indian data regulations. Scale infrastructure as the team grows.

Data parallelism replicates the model on each GPU and splits data batches. Tensor parallelism splits individual layers across GPUs (each GPU computes part of each layer). Pipeline parallelism splits layers across GPUs in stages. Data parallelism is simplest and scales well. Tensor parallelism is for layers too large for one GPU. Pipeline parallelism handles very deep models. Production systems often combine all three.

Use GPU-optimised inference servers (vLLM, TGI, or Triton) with continuous batching. Quantise the model to INT8/INT4 for speed. Implement KV cache efficiently. Deploy in regions close to users (Indian cloud regions). Use load balancing with health checks. Implement request queuing and timeout handling. Pre-warm model caches. Monitor P99 latency and auto-scale based on queue depth.

vLLM is a high-throughput LLM serving engine that introduces PagedAttention for efficient KV cache memory management. Like virtual memory in operating systems, it pages KV cache blocks, reducing memory waste from 60-80% to near zero. This enables 2-4x higher throughput than naive implementations. It also supports continuous batching and speculative decoding.

Use a streaming platform (Kafka, Kinesis) for event ingestion. Process events with a stream processor (Flink, Spark Streaming) for feature computation. Store computed features in a low-latency store (Redis, DynamoDB) for serving. Implement exactly-once processing semantics. Handle late-arriving data with watermarks. Monitor pipeline lag and throughput. Ensure consistency with batch-computed features.

Use model quantisation (INT8/INT4) to reduce weight memory. Implement efficient KV cache management (paged attention). Use model parallelism to split across GPUs. Implement dynamic batching to optimise memory utilisation. Use memory-efficient attention implementations (FlashAttention). Offload inactive model layers to CPU when possible. Monitor GPU memory with CUDA memory management tools.

Consider spot/preemptible GPU instances for training (60-90% savings). Use reserved instances for steady inference workloads. Implement job scheduling (SLURM, Kubernetes) for GPU sharing. Use gradient accumulation to use smaller GPUs effectively. Consider Indian cloud providers (Jio Cloud, Yotta) alongside AWS/GCP/Azure Mumbai regions. Set up auto-shutdown for idle resources. Monitor utilisation aggressively.

TensorRT is NVIDIA's inference optimisation SDK that takes trained models and produces highly optimised runtime engines. It applies layer fusion, kernel auto-tuning, precision calibration (FP16/INT8), and dynamic tensor memory management. It typically provides 2-5x speedup over framework-native inference. It is hardware-specific and requires re-optimisation for different GPU architectures.

High-bandwidth, low-latency interconnects are critical. NVLink connects GPUs within a node (900 GB/s on H100). InfiniBand or RoCE connects nodes (200-400 Gbps). Standard Ethernet (25-100 Gbps) is insufficient for large-scale training. Communication patterns (all-reduce, all-gather) dominate training time at scale. Place training nodes in the same rack or availability zone to minimise network latency.

Continuous batching (iteration-level batching) dynamically adds and removes requests from a batch at each generation step, unlike static batching which waits for all requests to complete. This prevents short requests from waiting for long ones, significantly improving GPU utilisation and throughput (2-10x). It is implemented in modern serving frameworks like vLLM and TGI.

Replicate model artifacts and configurations across regions. Use multi-region deployment with failover routing. Implement automated health checks and failover triggers. Back up model registries and feature stores. Document recovery procedures and test them regularly. Define RTO (Recovery Time Objective) and RPO (Recovery Point Objective). Keep model serving stateless for easy recovery.

Use distributed file systems (HDFS, Lustre) or object storage (S3) for training data. Implement data sharding across storage nodes. Use streaming data loaders that prefetch and pipeline IO with computation. Cache frequently accessed data on local NVMe SSDs. Optimise data format (TFRecord, WebDataset, Parquet) for sequential read performance. Monitor IO bottlenecks with storage metrics.

Deploy Triton or a custom serving layer on Kubernetes with GPU node pools in Indian cloud regions. Implement model routing based on request type. Use shared GPU scheduling to maximise utilisation across models. Implement a model registry for version management. Set up centralised monitoring and cost attribution per team. Handle compliance with Indian data regulations. Implement rate limiting and authentication per consumer.