How large language models work

An LLM is a neural network trained on massive text corpora to predict the next token in a sequence. It generates text autoregressively by sampling from a probability distribution over its vocabulary, one token at a time, using the previously generated tokens as context. Models like GPT-4 and Claude use the Transformer architecture.

Pre-training involves training a model on massive unlabelled text data with a self-supervised objective like next-token prediction. This gives the model general language understanding. Fine-tuning then adapts this pre-trained model to specific downstream tasks using smaller labelled datasets, transferring the knowledge learned during pre-training.

Zero-shot gives the model a task description without examples. One-shot includes one example of the expected input-output pair. Few-shot provides multiple examples to establish the pattern. Few-shot typically improves performance on complex or ambiguous tasks by helping the model understand the desired format and reasoning style through demonstration.

The system prompt sets the AI's behaviour, persona, and constraints across the conversation. User prompts contain the human's messages and queries. Assistant prompts contain the AI's responses and can be pre-filled to guide output format. This separation enables consistent behaviour while allowing dynamic user interactions. Most API providers support this three-role structure.

Temperature controls output randomness. Use low temperature (0-0.3) for factual, deterministic tasks like classification and extraction. Use higher temperature (0.7-1.0) for creative tasks like brainstorming. Top-p and top-k further shape the probability distribution. These parameters interact with prompt design; well-constrained prompts can tolerate higher temperatures.

Embeddings are dense, low-dimensional vector representations of high-dimensional data like words, images, or users. They capture semantic relationships so that similar items have nearby vectors. They enable efficient similarity search, are used in recommendation systems, and form the foundation of modern NLP models.

The Transformer uses self-attention mechanisms to process all tokens in parallel rather than sequentially like RNNs. This enables better long-range dependency capture and massive parallelisation during training. Key components include multi-head attention, positional encoding, feed-forward layers, and layer normalisation.

Attention computes a weighted sum of Value vectors, where the weights are determined by the compatibility between Query and Key vectors. For each token, Q represents what it is looking for, K represents what it offers, and V represents the actual content. The attention score is computed as softmax(QK^T / sqrt(d_k)) * V.

Multi-head attention runs several attention operations in parallel, each with different learned projection matrices. This allows the model to attend to information from different representation subspaces simultaneously. For example, one head might capture syntactic relationships while another captures semantic ones. The outputs are concatenated and linearly projected.

Tokenisation splits text into units (tokens) the model can process. Byte Pair Encoding (BPE) starts with individual characters and iteratively merges the most frequent adjacent pairs to create a vocabulary of subword units. This handles rare words by breaking them into known subwords, balancing vocabulary size with representation quality.

Since Transformers process all tokens in parallel (unlike RNNs), they have no inherent sense of token order. Positional encodings add information about each token's position in the sequence. The original Transformer used sinusoidal functions, while modern models often use learned positional embeddings or rotary positional embeddings (RoPE).

Encoder-only models like BERT process entire input bidirectionally and are great for classification and embeddings. Decoder-only models like GPT generate text autoregressively using causal masking. Encoder-decoder models like T5 use a full encoder for input understanding and a decoder for generation, excelling at translation and summarisation.

Temperature controls randomness by scaling logits before softmax; lower values make output more deterministic. Top-k sampling restricts selection to the k most probable tokens. Top-p (nucleus sampling) selects from the smallest set of tokens whose cumulative probability exceeds p. These parameters trade off between creativity and coherence.

The context window is the maximum number of tokens an LLM can process in a single forward pass. Extending it is challenging because self-attention has O(n^2) computational complexity and memory usage. Techniques like FlashAttention, sparse attention, sliding window attention, and RoPE scaling help extend context windows to 100K+ tokens.

Hallucination occurs when LLMs generate plausible-sounding but factually incorrect information. It happens because LLMs are trained to predict probable next tokens, not verified facts. Mitigation strategies include RAG for grounding in verified sources, chain-of-thought prompting, constrained decoding, and implementing fact-checking pipelines.

Causal language modelling (used by GPT) predicts the next token using only previous tokens, enforced by a causal attention mask. Masked language modelling (used by BERT) randomly masks tokens and predicts them using bidirectional context. Causal LM is suited for generation, while masked LM is better for understanding tasks.

In-context learning is the ability of LLMs to learn new tasks from examples provided in the prompt without updating model weights. Fine-tuning permanently modifies model weights through gradient descent on task-specific data. In-context learning is faster and more flexible but limited by context window size and generally less reliable for complex tasks.

Evaluate on task-specific benchmarks with your actual data, not just public leaderboards. Compare latency, throughput, cost per token, context window size, and output quality. Consider factors like API reliability, data privacy policies, and hosting options (cloud vs self-hosted). Run A/B tests and collect human evaluation scores on representative queries.

Greedy decoding always selects the most probable next token, which is fast but can produce repetitive text. Beam search maintains multiple candidate sequences and selects the highest-probability one, better for translation. Sampling-based methods (top-k, top-p) introduce controlled randomness for more creative and diverse outputs.

Cross-entropy loss measures the average negative log probability of correct tokens. Perplexity is the exponential of cross-entropy and represents the effective number of equally probable token choices the model considers. Lower perplexity indicates better prediction. Perplexity is more interpretable but both metrics are mathematically equivalent for comparison.

Key risks include generating harmful content, producing convincing misinformation, leaking private training data, being manipulated through prompt injection, encoding and amplifying biases, and enabling malicious uses like phishing. Mitigations include RLHF alignment, content filtering, red-teaming, and responsible deployment practices.

Vocabulary size determines how text is segmented. Larger vocabularies produce fewer tokens per text (saving context window) but increase embedding layer size and memory usage. Smaller vocabularies are more efficient memory-wise but create longer sequences. Typical sizes range from 32K to 128K tokens. Multilingual models need larger vocabularies to cover diverse scripts.

LLMs tend to pay more attention to information at the beginning and end of the context, potentially ignoring relevant information in the middle. Mitigate by placing the most relevant chunks first and last, limiting the number of retrieved chunks, using reranking to prioritise relevance, or restructuring retrieved content into a focused summary before generation.

Function calling is the mechanism by which LLMs generate structured tool invocations. OpenAI uses a tools parameter with JSON schema definitions. Anthropic (Claude) uses a similar tool_use format. Google Gemini has function declarations. The LLM outputs a structured call with function name and arguments rather than free text, which the application executes and returns results.

Structured output (JSON, function calls) ensures agent decisions are parseable and actionable by the orchestration system. Without structure, the system must extract tool calls and parameters from free text, which is error-prone. Modern LLM APIs support constrained output modes that guarantee valid JSON. This reliability is essential for production agent systems.

SFT trains the model to mimic demonstration data by minimising the difference between its outputs and the training examples. RLHF adds a preference optimisation step where a reward model learns human preferences and the LLM is optimised to maximise that reward using reinforcement learning. SFT teaches what to generate; RLHF teaches what humans prefer among options.

MMLU (Massive Multitask Language Understanding) evaluates LLMs across 57 subjects from STEM, humanities, and social sciences at varying difficulty levels. It measures a model's breadth of knowledge and reasoning through multiple-choice questions. While widely used for model comparison, it has limitations including potential training data contamination and not measuring generation quality.

Reinforcement Learning from Human Feedback trains a reward model on human preferences about response quality, then uses PPO or similar algorithms to fine-tune the LLM to maximise that reward. It aligns model outputs with human values, making responses more helpful, harmless, and honest compared to pure supervised fine-tuning.

KV cache stores the key and value tensors computed for previously processed tokens so they do not need to be recomputed at each generation step. This reduces inference time from O(n^2) to O(n) per token. However, it requires significant GPU memory, especially for long sequences and large batch sizes.

In FP16, each parameter takes 2 bytes, so 70B parameters need approximately 140 GB just for model weights. Add KV cache memory which depends on context length and batch size. A rough estimate for serving would be 160-200 GB, requiring multiple A100 (80GB) or H100 GPUs with tensor parallelism. Quantisation to INT4 can reduce this to around 40 GB.

Emergent abilities are capabilities that appear only above certain model scale thresholds and are not present in smaller models. Examples include in-context learning, chain-of-thought reasoning, and few-shot arithmetic. These are debated in the research community, with some arguing they may be artefacts of evaluation metrics rather than true phase transitions.

Quantisation reduces model weights from higher precision (FP32/FP16) to lower precision (INT8/INT4), dramatically reducing memory usage and improving inference speed. Trade-offs include slight quality degradation, especially for complex reasoning tasks. Techniques like GPTQ, AWQ, and GGML enable 4-bit quantisation with minimal quality loss for many applications.

Scaling laws (Kaplan et al., Chinchilla) describe how model performance improves predictably as a power law of model size, dataset size, and compute budget. Chinchilla showed that models should be trained on roughly 20 tokens per parameter for compute-optimal training. These laws help estimate the resources needed for a desired performance level.

FlashAttention is an IO-aware exact attention algorithm that reduces memory reads/writes by tiling the computation and keeping intermediate results in fast SRAM. It achieves 2-4x speedup in wall-clock time and reduces memory usage from O(n^2) to O(n) without approximating the attention computation. It has become standard in modern LLM training and inference.

Speculative decoding uses a smaller draft model to generate candidate tokens quickly, then the larger target model verifies them in a single forward pass. If the draft tokens match what the target would have generated, multiple tokens are accepted at once. This can achieve 2-3x speedup without changing the output distribution.

MoE models contain multiple expert sub-networks and a gating network that routes each token to a subset of experts. Only a fraction of parameters are activated per token, enabling much larger total parameter counts with manageable compute costs. Mixtral 8x7B, for example, has 47B total parameters but only activates 13B per token.

Choose a model with strong multilingual capabilities (like GPT-4, Claude, or IndicBERT for embedding). Implement language detection for routing. Use RAG with a knowledge base containing Indian language content. Fine-tune on Indian language conversation data if quality is insufficient. Handle transliteration (Hindi in Roman script) and code-mixing (Hinglish) explicitly.

Distillation trains a smaller student model to mimic the outputs of a larger teacher model, resulting in a genuinely smaller architecture. Quantisation keeps the same architecture but reduces the numerical precision of weights. Distillation requires retraining but can achieve better compression ratios; quantisation is faster to apply but limited in compression.

DPO directly optimises the LLM policy using preference pairs without training a separate reward model, simplifying the RLHF pipeline. It reformulates the reward-based objective as a classification loss on preference data. DPO is simpler to implement and more stable but may not perform as well as RLHF with a well-tuned reward model on complex alignment tasks.

Constitutional AI uses a set of principles (a constitution) to guide model behaviour through self-critique and revision. The model generates responses, critiques them against the constitution, and revises accordingly. In prompt engineering, similar principles can be embedded in system prompts to create self-correcting behaviour without explicit human feedback on each response.

RLAIF (RL from AI Feedback) replaces human feedback with AI-generated feedback based on a constitution (set of principles). An AI model critiques and revises outputs according to these principles, generating preference data. This preference data trains a reward model or directly optimises the policy. It is more scalable than human feedback while maintaining alignment quality.

Knowledge distillation trains a smaller student model to mimic a larger teacher model's output distributions, not just hard labels. The student learns the teacher's soft probabilities, which contain richer information. It is used to create efficient production models. Combined with fine-tuning, you can distill a large fine-tuned model into a smaller, deployable one with minimal quality loss.

DPO simplifies preference-based fine-tuning by eliminating the need for a separate reward model. It directly optimises the policy model using pairs of preferred and rejected responses. The loss function implicitly defines a reward and trains the model to increase the probability of preferred responses while decreasing rejected ones. It is simpler and more stable than PPO-based RLHF.

vLLM is a high-throughput LLM serving engine that introduces PagedAttention for efficient KV cache memory management. Like virtual memory in operating systems, it pages KV cache blocks, reducing memory waste from 60-80% to near zero. This enables 2-4x higher throughput than naive implementations. It also supports continuous batching and speculative decoding.

Continuous batching (iteration-level batching) dynamically adds and removes requests from a batch at each generation step, unlike static batching which waits for all requests to complete. This prevents short requests from waiting for long ones, significantly improving GPU utilisation and throughput (2-10x). It is implemented in modern serving frameworks like vLLM and TGI.