AI infrastructure and cost

AI infrastructure is the computing layer that makes AI work — the GPUs that train and run models, the serving systems that handle prediction requests, the storage for datasets and model weights, and the networking that connects it all. It is backend engineering and DevOps, viewed through the lens of AI workloads.

AI workloads are different from traditional web services. They are GPU-intensive rather than CPU-intensive. A single inference request might take 10–100x more compute than a typical API call. Batch training jobs can run for hours or days. This changes how you architect, scale and optimise everything.

GPUs (Graphics Processing Units) are used for AI because they can perform thousands of matrix operations in parallel — and neural networks are fundamentally matrix operations. NVIDIA dominates the AI GPU market with cards like the A100, H100 and H200.

GPU memory (VRAM) is usually the bottleneck. A large language model needs its entire weights in GPU memory to run. A 70-billion parameter model in full precision needs about 140 GB of VRAM — more than a single GPU has. This is why quantisation (reducing precision) and model parallelism (splitting across multiple GPUs) matter.

Running a model in production (inference) is where most of the ongoing cost lives. Optimisation techniques include: quantisation (4-bit models run 4x faster with small quality loss), KV-caching (avoiding redundant computation for sequential tokens), batching (processing multiple requests together), speculative decoding (using a smaller model to speed up a larger one), and choosing the right hardware for your workload.

For LLM inference specifically, the key metrics are time-to-first-token (TTFT) and tokens-per-second. Users notice TTFT more than total generation time.

AI infrastructure costs can spiral quickly. A single H100 GPU costs about $30K, and cloud rental is $2–4 per GPU-hour. A company running a large LLM for thousands of concurrent users can easily spend $50K–500K per month on inference alone.

Cost optimisation strategies include: using the smallest model that meets quality requirements, quantising aggressively, batching requests, caching common queries, using spot/preemptible instances for training, and routing simple queries to cheaper models.

AI infrastructure engineering is where backend engineering, DevOps and cloud meet AI. If you have experience with Docker, Kubernetes, cloud providers, system performance and monitoring, you already have 70% of what this role needs. Add GPU management, model serving and inference optimisation, and you are an AI infrastructure engineer.

This role is chronically understaffed because it requires both traditional infrastructure skills and AI-specific knowledge. Most pure ML engineers do not want to manage infrastructure, and most DevOps engineers have not learned AI workloads yet.