AI infrastructure and cost

A vector database stores and indexes high-dimensional vector embeddings for efficient similarity search. Unlike traditional databases that use exact matching on structured fields, vector databases use approximate nearest neighbour (ANN) algorithms like HNSW or IVF for fast similarity search. Popular options include Pinecone, Weaviate, Qdrant, ChromaDB, and pgvector.

Docker containers package the model, code, dependencies, and runtime into a portable, reproducible unit. This eliminates 'works on my machine' issues, ensures consistent behaviour across environments, simplifies deployment, and enables scaling via orchestrators like Kubernetes. Use multi-stage builds to minimise image size and separate training from serving containers.

GPUs (NVIDIA A100, H100) are the dominant hardware for training and inference due to their massive parallelism. TPUs (Google's custom chips) are optimised for tensor operations. CPUs handle preprocessing and lightweight inference. Specialised accelerators like AWS Inferentia target inference cost-efficiency. FPGAs offer programmable acceleration for specific workloads.

CUDA is NVIDIA's parallel computing platform that enables general-purpose GPU programming. It provides APIs for launching parallel computations on GPU cores. Deep learning frameworks (PyTorch, TensorFlow) use CUDA internally for accelerated tensor operations. CUDA's ecosystem including cuBLAS, cuDNN, and NCCL is the primary reason NVIDIA dominates the AI hardware market.

Vertical scaling adds more resources (bigger GPU, more memory) to a single instance, limited by hardware maximums. Horizontal scaling adds more instances behind a load balancer, limited by coordination complexity. For ML serving, use vertical scaling for model size constraints (larger model needs bigger GPU) and horizontal scaling for throughput. Most production systems combine both approaches.

Implement per-client rate limiting using token buckets or sliding window counters. Return appropriate error codes (429) with retry-after headers. Track usage per authenticated user. Set tool-specific rate limits based on the cost of the underlying operation. Communicate limits through tool descriptions. Cache frequent requests. Consider implementing priority queues for different user tiers.

Agent loops consume many LLM tokens through repeated reasoning, tool calls, and observations. Costs multiply with retries, long context, and multi-agent coordination. Optimise by using cheaper models for simple tool selection, caching frequent operations, limiting iteration counts, compressing context, and batching similar tasks. Monitor cost per completed task and set budgets per workflow.

For a 7B model: QLoRA needs ~16GB VRAM (single consumer GPU), full fine-tuning needs ~112GB (2x A100). For 13B: QLoRA needs ~24GB, full needs ~208GB. For 70B: QLoRA needs ~48GB (single A100), full needs ~1TB+ (multi-node). Training time scales roughly linearly with data size. Use gradient checkpointing and DeepSpeed ZeRO to reduce memory at the cost of speed.

Measure P50, P95, and P99 latency at each pipeline stage (preprocessing, model inference, postprocessing). Profile with tools like NVIDIA Nsight for GPU utilisation. Optimise by reducing model size (quantisation, distillation), batching requests, using faster hardware, implementing caching, and optimising tokenisation. Set latency SLAs based on user experience requirements.

Kubeflow is an open-source ML platform built on Kubernetes that provides components for the full ML lifecycle. It includes Kubeflow Pipelines for workflow orchestration, KFServing for model serving, Katib for hyperparameter tuning, and notebook servers for development. It enables scalable, reproducible ML workflows on any Kubernetes cluster, popular in Indian enterprises using on-premise infrastructure.

Model compression reduces model size and inference cost for production deployment. Techniques include quantisation (reducing precision from FP32 to INT8/INT4), pruning (removing unimportant weights or neurons), knowledge distillation (training a smaller model), and architecture search (finding efficient architectures). Choose based on acceptable quality loss and target deployment hardware.

Tag all ML resources for cost attribution. Use spot/preemptible instances for training (60-90% savings). Right-size GPU instances based on actual utilisation. Implement auto-scaling for serving. Use model optimisation (quantisation, distillation) to reduce inference costs. Schedule batch jobs during off-peak hours. Set up budget alerts and cost anomaly detection. Review costs weekly.

The H100 offers approximately 3x the performance of A100 for transformer training thanks to the Transformer Engine with FP8 support, higher memory bandwidth (3.35 TB/s vs 2 TB/s), and more CUDA cores. H100 has 80GB HBM3 memory vs A100's 80GB HBM2e. H100 also introduces NVLink 4.0 for faster multi-GPU communication. H100 is preferred for large model training and inference.

Cloud offers flexibility, scalability, and no upfront capital. On-premise provides lower long-term costs for sustained workloads and data sovereignty. Cloud is better for experimentation, burst capacity, and teams without infrastructure expertise. On-premise makes sense for consistent high utilisation (>70%), sensitive data requirements (Indian data localisation), or large-scale training where reserved instances cost more than owned hardware.

Profile GPU utilisation with nvidia-smi and PyTorch profiler. Increase batch size to fill GPU memory. Use mixed precision training (FP16/BF16) for higher throughput. Optimise data loading with multiple workers and prefetching. Enable gradient checkpointing to trade compute for memory. Use DeepSpeed or FSDP for multi-GPU efficiency. Reduce CPU-GPU data transfer bottlenecks.

Define scaling metrics: GPU utilisation, request queue depth, latency percentiles, or custom business metrics. Set target values and scaling thresholds. Use Kubernetes HPA (Horizontal Pod Autoscaler) or cloud auto-scaling groups. Implement warm-up time for new instances (model loading takes time). Set minimum replicas for baseline traffic. Use predictive scaling for known traffic patterns.

Evaluate: query performance (latency at scale), scalability (data size and QPS), index types supported (HNSW, IVF), filtering capabilities (metadata filters), managed vs self-hosted, cost model, backup and replication, cloud region availability (Indian regions), API compatibility with your stack, and community/support quality. Popular choices include Pinecone, Weaviate, Qdrant, Milvus, and pgvector.

ONNX (Open Neural Network Exchange) is an open format for representing ML models, enabling interoperability between frameworks. You can train in PyTorch and deploy with ONNX Runtime, TensorRT, or OpenVINO. ONNX Runtime provides optimised inference across CPU, GPU, and edge devices. This decouples training framework choice from deployment infrastructure, enabling hardware-specific optimisations.

Kubernetes orchestrates containerised applications across clusters of machines. For ML, it provides GPU scheduling and sharing, auto-scaling of inference services, resource isolation for multiple teams, rolling deployments with zero downtime, self-healing when containers fail, and infrastructure-as-code for reproducibility. It is the standard platform for production ML serving.

Use PyTorch's DistributedDataParallel (DDP) with NCCL backend for inter-GPU communication. Set up process groups with one process per GPU. Use DistributedSampler for data partitioning. Launch with torchrun specifying nproc_per_node. Enable mixed precision with AMP. Ensure batch size is per-GPU and adjust learning rate for effective global batch size. Profile to verify GPU utilisation.

Edge deployment runs AI models on devices near the data source (phones, IoT devices, local servers) rather than in the cloud. It is appropriate when low latency is critical, network connectivity is unreliable (rural India), data privacy prevents cloud transmission, or cloud costs are prohibitive at scale. Requires model compression (quantisation, pruning) for device constraints.

Use object storage (S3, GCS, Azure Blob) as the foundation. Implement a medallion architecture: bronze (raw data), silver (cleaned and validated), gold (feature-ready). Use Delta Lake or Iceberg for ACID transactions and time travel. Catalog data with Apache Hive or Unity Catalog. Implement access controls and data lineage tracking. Optimise with columnar formats (Parquet) and partitioning.

Cache at multiple levels: model weights in GPU memory (avoid reloading), KV cache for conversation history, query-result cache for frequent queries (Redis/Memcached), and embedding cache for common inputs. Implement cache invalidation based on model version changes. Use LRU eviction for memory-bounded caches. Monitor cache hit rates and adjust sizing accordingly.

Use DCGM (Data Center GPU Manager) for GPU metrics. Monitor GPU utilisation, memory usage, temperature, power consumption, and ECC errors. Integrate with Prometheus for collection and Grafana for visualisation. Set alerts for low utilisation (wasted resources), high temperature (throttling risk), and ECC errors (hardware failure). Track GPU time per team for chargeback. Monitor driver and CUDA version consistency.

Triton is an open-source inference serving platform that supports multiple frameworks (PyTorch, TensorFlow, ONNX, TensorRT) simultaneously. Key features include dynamic batching, model versioning, model ensembles, GPU scheduling across multiple models, metrics reporting, and HTTP/gRPC APIs. It is production-ready and widely used in enterprise deployments for serving multiple models efficiently.

Serverless inference (AWS Lambda, Google Cloud Functions, Azure Functions) auto-scales to zero when idle and scales up on demand. It eliminates infrastructure management and charges only for actual usage. Appropriate for low-traffic or bursty ML endpoints with small models. Not suitable for large models (memory limits), GPU workloads, or latency-sensitive applications due to cold start times.

KV cache stores the key and value tensors computed for previously processed tokens so they do not need to be recomputed at each generation step. This reduces inference time from O(n^2) to O(n) per token. However, it requires significant GPU memory, especially for long sequences and large batch sizes.

In FP16, each parameter takes 2 bytes, so 70B parameters need approximately 140 GB just for model weights. Add KV cache memory which depends on context length and batch size. A rough estimate for serving would be 160-200 GB, requiring multiple A100 (80GB) or H100 GPUs with tensor parallelism. Quantisation to INT4 can reduce this to around 40 GB.

Quantisation reduces model weights from higher precision (FP32/FP16) to lower precision (INT8/INT4), dramatically reducing memory usage and improving inference speed. Trade-offs include slight quality degradation, especially for complex reasoning tasks. Techniques like GPTQ, AWQ, and GGML enable 4-bit quantisation with minimal quality loss for many applications.

FlashAttention is an IO-aware exact attention algorithm that reduces memory reads/writes by tiling the computation and keeping intermediate results in fast SRAM. It achieves 2-4x speedup in wall-clock time and reduces memory usage from O(n^2) to O(n) without approximating the attention computation. It has become standard in modern LLM training and inference.

Speculative decoding uses a smaller draft model to generate candidate tokens quickly, then the larger target model verifies them in a single forward pass. If the draft tokens match what the target would have generated, multiple tokens are accepted at once. This can achieve 2-3x speedup without changing the output distribution.

Use Apache Kafka or Redis Streams for message ingestion, asyncio or Faust for stream processing, and a FastAPI/gRPC endpoint for inference. Implement windowed feature computation with a sliding buffer. Use connection pooling for database lookups and batch inference requests when possible. Monitor latency and throughput with Prometheus metrics.

Mixed precision uses FP16 for most operations and FP32 for numerically sensitive operations, reducing memory usage by nearly half and speeding up training on modern GPUs. In PyTorch, use torch.cuda.amp.autocast() for the forward pass and torch.cuda.amp.GradScaler for gradient scaling to prevent underflow. This typically yields 1.5-2x speedup.

Use torch.nn.parallel.DistributedDataParallel (DDP) for data parallelism. Initialise process groups with torch.distributed.init_process_group. Wrap the model in DDP and use DistributedSampler for the DataLoader. Launch with torchrun or torch.distributed.launch. For model parallelism across GPUs, use pipeline parallelism or tensor parallelism with libraries like DeepSpeed or FSDP.

Store access control metadata (tenant, department, role) with each chunk. Apply metadata filters at query time to enforce permissions. Integrate with the enterprise's existing IAM (like Active Directory). Ensure vector database supports metadata filtering efficiently. Implement audit logging for compliance. Handle Indian data localisation requirements by keeping sensitive data in India-based storage.

Cache frequent queries and their results. Use approximate nearest neighbour indices for faster search. Implement query embedding caching. Batch vector lookups. Use streaming for LLM generation. Pre-compute and store embeddings for static content. Optimise chunk sizes to reduce the number of retrievals. Consider using smaller, faster models for retrieval stages.

Create separate MCP servers for each domain: HR (employee data, leave management), Finance (invoicing, expense reports), CRM (customer data), and IT (ticket management). Implement a gateway server for authentication and routing. Use SSE transport for shared remote servers. Standardise tool naming and error handling across servers. Document tools for both LLMs and developers.

Use a persistent state store (database or Redis) for workflow state. Implement checkpointing at each step so workflows can resume after failures. Serialise state in a structured format with version tracking. Handle concurrent access with locking mechanisms. Implement state cleanup for abandoned workflows. Use event sourcing for full auditability of state changes.

Stream the LLM's reasoning tokens as they are generated for transparency. Send tool execution status updates as events. Implement partial result streaming for long operations. Use WebSockets or SSE for the client connection. Display thinking indicators and progress bars. Allow users to interrupt and redirect the agent mid-execution. Buffer events for connection resilience.

Equip the agent with tools for monitoring (Grafana/CloudWatch queries), deployment (kubectl, CI/CD triggers), log analysis, and alert management. Implement strict approval workflows for production changes. Start with read-only tools and gradually add write capabilities with human approval. Handle Indian cloud regions (Mumbai, Hyderabad). Include incident response runbooks as context. Monitor agent actions carefully during the initial rollout period.

QLoRA combines quantisation with LoRA. It loads the base model in 4-bit precision (NormalFloat4) and adds LoRA adapters in higher precision. This reduces memory requirements by approximately 4x compared to LoRA alone, enabling fine-tuning of 65B parameter models on a single 48GB GPU. It uses techniques like double quantisation and paged optimisers for memory efficiency.

Use a model serving framework (TorchServe, Triton, or TF Serving) behind a load balancer. Deploy in Indian cloud regions (AWS Mumbai, Azure Central India) for low latency. Implement auto-scaling based on traffic patterns (handle sale events like Diwali/Republic Day). Use model caching and batching for efficiency. Set up health checks, monitoring, and automatic failover. Consider edge deployment for latency-critical features.

Start with managed services to move fast: GitHub for code, DVC for data versioning, MLflow for experiment tracking, AWS SageMaker or GCP Vertex AI for training and serving. Implement CI/CD with GitHub Actions. Use Docker and Kubernetes for serving. Set up monitoring with Prometheus and Grafana. Automate data validation and model evaluation. Plan for compliance with Indian data regulations. Scale infrastructure as the team grows.

Data parallelism replicates the model on each GPU and splits data batches. Tensor parallelism splits individual layers across GPUs (each GPU computes part of each layer). Pipeline parallelism splits layers across GPUs in stages. Data parallelism is simplest and scales well. Tensor parallelism is for layers too large for one GPU. Pipeline parallelism handles very deep models. Production systems often combine all three.

Use GPU-optimised inference servers (vLLM, TGI, or Triton) with continuous batching. Quantise the model to INT8/INT4 for speed. Implement KV cache efficiently. Deploy in regions close to users (Indian cloud regions). Use load balancing with health checks. Implement request queuing and timeout handling. Pre-warm model caches. Monitor P99 latency and auto-scale based on queue depth.

vLLM is a high-throughput LLM serving engine that introduces PagedAttention for efficient KV cache memory management. Like virtual memory in operating systems, it pages KV cache blocks, reducing memory waste from 60-80% to near zero. This enables 2-4x higher throughput than naive implementations. It also supports continuous batching and speculative decoding.

Use a streaming platform (Kafka, Kinesis) for event ingestion. Process events with a stream processor (Flink, Spark Streaming) for feature computation. Store computed features in a low-latency store (Redis, DynamoDB) for serving. Implement exactly-once processing semantics. Handle late-arriving data with watermarks. Monitor pipeline lag and throughput. Ensure consistency with batch-computed features.

DeepSpeed is a deep learning optimisation library from Microsoft that enables training models with billions of parameters. Its ZeRO (Zero Redundancy Optimiser) partitions model states across GPUs to reduce memory per GPU. It supports 3D parallelism (data, tensor, pipeline), mixed precision, gradient checkpointing, and offloading to CPU/NVMe. It can train trillion-parameter models on clusters of GPUs.

Use model quantisation (INT8/INT4) to reduce weight memory. Implement efficient KV cache management (paged attention). Use model parallelism to split across GPUs. Implement dynamic batching to optimise memory utilisation. Use memory-efficient attention implementations (FlashAttention). Offload inactive model layers to CPU when possible. Monitor GPU memory with CUDA memory management tools.

Consider spot/preemptible GPU instances for training (60-90% savings). Use reserved instances for steady inference workloads. Implement job scheduling (SLURM, Kubernetes) for GPU sharing. Use gradient accumulation to use smaller GPUs effectively. Consider Indian cloud providers (Jio Cloud, Yotta) alongside AWS/GCP/Azure Mumbai regions. Set up auto-shutdown for idle resources. Monitor utilisation aggressively.

TensorRT is NVIDIA's inference optimisation SDK that takes trained models and produces highly optimised runtime engines. It applies layer fusion, kernel auto-tuning, precision calibration (FP16/INT8), and dynamic tensor memory management. It typically provides 2-5x speedup over framework-native inference. It is hardware-specific and requires re-optimisation for different GPU architectures.

High-bandwidth, low-latency interconnects are critical. NVLink connects GPUs within a node (900 GB/s on H100). InfiniBand or RoCE connects nodes (200-400 Gbps). Standard Ethernet (25-100 Gbps) is insufficient for large-scale training. Communication patterns (all-reduce, all-gather) dominate training time at scale. Place training nodes in the same rack or availability zone to minimise network latency.

Continuous batching (iteration-level batching) dynamically adds and removes requests from a batch at each generation step, unlike static batching which waits for all requests to complete. This prevents short requests from waiting for long ones, significantly improving GPU utilisation and throughput (2-10x). It is implemented in modern serving frameworks like vLLM and TGI.

Replicate model artifacts and configurations across regions. Use multi-region deployment with failover routing. Implement automated health checks and failover triggers. Back up model registries and feature stores. Document recovery procedures and test them regularly. Define RTO (Recovery Time Objective) and RPO (Recovery Point Objective). Keep model serving stateless for easy recovery.

Use distributed file systems (HDFS, Lustre) or object storage (S3) for training data. Implement data sharding across storage nodes. Use streaming data loaders that prefetch and pipeline IO with computation. Cache frequently accessed data on local NVMe SSDs. Optimise data format (TFRecord, WebDataset, Parquet) for sequential read performance. Monitor IO bottlenecks with storage metrics.

Deploy Triton or a custom serving layer on Kubernetes with GPU node pools in Indian cloud regions. Implement model routing based on request type. Use shared GPU scheduling to maximise utilisation across models. Implement a model registry for version management. Set up centralised monitoring and cost attribution per team. Handle compliance with Indian data regulations. Implement rate limiting and authentication per consumer.