NLP Engineer

An LLM is a neural network trained on massive text corpora to predict the next token in a sequence. It generates text autoregressively by sampling from a probability distribution over its vocabulary, one token at a time, using the previously generated tokens as context. Models like GPT-4 and Claude use the Transformer architecture.

Pre-training involves training a model on massive unlabelled text data with a self-supervised objective like next-token prediction. This gives the model general language understanding. Fine-tuning then adapts this pre-trained model to specific downstream tasks using smaller labelled datasets, transferring the knowledge learned during pre-training.

Cosine similarity measures the angle between vectors (direction), ranging from -1 to 1. Euclidean distance measures the straight-line distance between points (magnitude matters). Use cosine similarity when only direction matters (text embeddings, recommendation systems). Use Euclidean distance when magnitude is meaningful (spatial data, anomaly detection based on deviation).

Prompt engineering is the practice of designing and refining inputs to LLMs to elicit desired outputs. It matters because LLMs are highly sensitive to how instructions are framed. Good prompts can dramatically improve output quality, accuracy, and relevance without any model changes, making it a cost-effective alternative to fine-tuning for many use cases.

Zero-shot gives the model a task description without examples. One-shot includes one example of the expected input-output pair. Few-shot provides multiple examples to establish the pattern. Few-shot typically improves performance on complex or ambiguous tasks by helping the model understand the desired format and reasoning style through demonstration.

RAG combines information retrieval with LLM generation. It retrieves relevant documents from a knowledge base and includes them in the prompt context, allowing the LLM to generate answers grounded in specific data. RAG reduces hallucination, enables access to private/current data without fine-tuning, and is more cost-effective than retraining models.

Fine-tuning adapts a pre-trained model to a specific task by continuing training on task-specific data. Use it when prompt engineering alone is insufficient, you need consistent output formatting, domain-specific knowledge integration, or significant behaviour changes. It is more expensive than prompting but cheaper than training from scratch.

Transfer learning involves taking a model pre-trained on a large dataset and fine-tuning it on a smaller, task-specific dataset. It reduces training time and data requirements dramatically. Models like BERT and ResNet are commonly used as pre-trained bases for NLP and computer vision tasks.

Embeddings are dense, low-dimensional vector representations of high-dimensional data like words, images, or users. They capture semantic relationships so that similar items have nearby vectors. They enable efficient similarity search, are used in recommendation systems, and form the foundation of modern NLP models.

The Transformer uses self-attention mechanisms to process all tokens in parallel rather than sequentially like RNNs. This enables better long-range dependency capture and massive parallelisation during training. Key components include multi-head attention, positional encoding, feed-forward layers, and layer normalisation.

Attention computes a weighted sum of Value vectors, where the weights are determined by the compatibility between Query and Key vectors. For each token, Q represents what it is looking for, K represents what it offers, and V represents the actual content. The attention score is computed as softmax(QK^T / sqrt(d_k)) * V.

Multi-head attention runs several attention operations in parallel, each with different learned projection matrices. This allows the model to attend to information from different representation subspaces simultaneously. For example, one head might capture syntactic relationships while another captures semantic ones. The outputs are concatenated and linearly projected.

Tokenisation splits text into units (tokens) the model can process. Byte Pair Encoding (BPE) starts with individual characters and iteratively merges the most frequent adjacent pairs to create a vocabulary of subword units. This handles rare words by breaking them into known subwords, balancing vocabulary size with representation quality.

Since Transformers process all tokens in parallel (unlike RNNs), they have no inherent sense of token order. Positional encodings add information about each token's position in the sequence. The original Transformer used sinusoidal functions, while modern models often use learned positional embeddings or rotary positional embeddings (RoPE).

Encoder-only models like BERT process entire input bidirectionally and are great for classification and embeddings. Decoder-only models like GPT generate text autoregressively using causal masking. Encoder-decoder models like T5 use a full encoder for input understanding and a decoder for generation, excelling at translation and summarisation.

Temperature controls randomness by scaling logits before softmax; lower values make output more deterministic. Top-k sampling restricts selection to the k most probable tokens. Top-p (nucleus sampling) selects from the smallest set of tokens whose cumulative probability exceeds p. These parameters trade off between creativity and coherence.

Hallucination occurs when LLMs generate plausible-sounding but factually incorrect information. It happens because LLMs are trained to predict probable next tokens, not verified facts. Mitigation strategies include RAG for grounding in verified sources, chain-of-thought prompting, constrained decoding, and implementing fact-checking pipelines.

Causal language modelling (used by GPT) predicts the next token using only previous tokens, enforced by a causal attention mask. Masked language modelling (used by BERT) randomly masks tokens and predicts them using bidirectional context. Causal LM is suited for generation, while masked LM is better for understanding tasks.

In-context learning is the ability of LLMs to learn new tasks from examples provided in the prompt without updating model weights. Fine-tuning permanently modifies model weights through gradient descent on task-specific data. In-context learning is faster and more flexible but limited by context window size and generally less reliable for complex tasks.

Greedy decoding always selects the most probable next token, which is fast but can produce repetitive text. Beam search maintains multiple candidate sequences and selects the highest-probability one, better for translation. Sampling-based methods (top-k, top-p) introduce controlled randomness for more creative and diverse outputs.

Cross-entropy loss measures the average negative log probability of correct tokens. Perplexity is the exponential of cross-entropy and represents the effective number of equally probable token choices the model considers. Lower perplexity indicates better prediction. Perplexity is more interpretable but both metrics are mathematically equivalent for comparison.

Vocabulary size determines how text is segmented. Larger vocabularies produce fewer tokens per text (saving context window) but increase embedding layer size and memory usage. Smaller vocabularies are more efficient memory-wise but create longer sequences. Typical sizes range from 32K to 128K tokens. Multilingual models need larger vocabularies to cover diverse scripts.

Define the role (banking support agent), scope (account queries, transactions, card services), tone (professional, helpful, in English and Hindi). Include guardrails against providing financial advice or sharing account details in responses. Specify escalation paths for complaints. Include context about Indian banking terminology (UPI, NEFT, RTGS) and RBI regulations relevant to customer interactions.

Provide 3-5 diverse examples per category covering Indian telecom issues (network outages, billing disputes, SIM-related, data pack queries, number portability). Include edge cases and examples in both English and Hinglish. Structure examples with clear input-output pairs. Add instructions for ambiguous cases and a fallback 'Other' category. Test with regional variations.

Prompt engineering crafts inputs to guide a model's existing capabilities without changing weights. Instruction tuning fine-tunes the model on instruction-following data to make it inherently better at following prompts. Instruction-tuned models (ChatGPT, Claude) respond better to natural language prompts compared to base models that require more carefully engineered prompts.

Reranking applies a more expensive cross-encoder model to rescore initially retrieved documents based on their relevance to the query. The initial retrieval uses fast bi-encoder embeddings to get candidate documents, then the reranker (like Cohere Rerank or a cross-encoder) scores each query-document pair jointly. This two-stage approach balances speed and accuracy.

Dense retrieval uses learned vector embeddings from neural models to represent queries and documents as continuous vectors, capturing semantic similarity. Sparse retrieval uses traditional methods like BM25 with term frequency and inverse document frequency. Dense is better for semantic matching; sparse excels at exact term matching. Combining both often yields the best results.

Use RAG when you need access to frequently updated data, require citations, or have a large knowledge base. Use fine-tuning when you need to change the model's behaviour, style, or reasoning patterns, or when knowledge is stable and fits in training data. RAG is cheaper and more flexible; fine-tuning gives better performance for specific formats. Many production systems combine both approaches.

Compare against known ground truth databases or verified sources. Use claim decomposition to break responses into individual facts and verify each. Implement automated fact-checking pipelines using knowledge bases. Use NLI (natural language inference) models to check consistency between claims and source documents. Track hallucination rate as a key metric. Human spot-checking remains essential for novel claims.

BLEU only measures n-gram overlap and misses semantic quality. Better approaches include BERTScore (embedding similarity), METEOR (semantic matching with synonyms), human evaluation along dimensions like fluency, coherence, and relevance, LLM-as-judge scoring, and task-specific metrics. For summarisation, use ROUGE and factual consistency checks. Combine multiple metrics for a comprehensive view.

Choose a model with strong multilingual capabilities (like GPT-4, Claude, or IndicBERT for embedding). Implement language detection for routing. Use RAG with a knowledge base containing Indian language content. Fine-tune on Indian language conversation data if quality is insufficient. Handle transliteration (Hindi in Roman script) and code-mixing (Hinglish) explicitly.

Break extraction into stages: first identify document type (Aadhaar, PAN, invoice), then extract specific fields per type. Use structured output format with field-level confidence scores. Handle multiple Indian scripts and transliterations. Provide examples of each document type. Include validation rules (PAN format: ABCDE1234F, Aadhaar: 12 digits). Chain extraction and validation prompts.

Chain 1: Extract structured data (name, education, skills, experience, certifications). Chain 2: Score against job requirements (tech stack match, years of experience, education level). Chain 3: Generate a summary highlighting strengths and gaps. Include Indian education system context (IIT/NIT/IIIT tiers, GATE scores, Indian certifications). Handle varied resume formats and Indianised English.

Use the target language for examples and instructions when the model supports it. Handle code-mixing (Hinglish, Tanglish) explicitly. Test tokenisation efficiency for non-Latin scripts. Provide transliterated alternatives. Use language-specific formatting conventions. Test output quality in each language separately and adjust temperature. Consider language-specific evaluation metrics.

Specify the legal domain (corporate, criminal, tax under Indian statutes). Instruct the model to preserve legal terminology and statutory references accurately. Include disclaimers that summaries are not legal advice. Structure output with key findings, relevant sections/acts, and actionable items. Handle references to Indian acts (IT Act, Companies Act) and court hierarchies. Test against verified legal summaries.

Ingest PDFs of policy documents with structure-aware parsing for Indian government formatting. Handle Hindi and English content with multilingual embeddings. Implement metadata filtering by ministry, date, and policy type. Use hybrid search for exact policy numbers and semantic queries. Include citation generation with specific section references. Handle amendments and superseded policies.

Collect domain-specific query-document pairs from search logs or manually curated examples. Create training triplets (query, positive document, negative document) using hard negative mining. Fine-tune a pre-trained embedding model (like BGE or E5) using contrastive loss. Evaluate on a held-out domain-specific test set using retrieval metrics. This typically yields significant improvement over general embeddings.

Equip the agent with tools for order lookup, return processing, refund status, and FAQ search. Support multilingual interactions (Hindi, English, and regional languages). Implement escalation to human agents for complex issues. Include tools for checking shipping status with Indian logistics providers (Delhivery, BlueDart). Handle Indian-specific payment issues (UPI failures, COD returns). Add sentiment detection for priority routing.

Create specialised agents: a classifier agent for categorising content type, a policy checker agent with access to moderation guidelines, a context analyser for understanding cultural nuances (important for Indian content), and a decision agent that synthesises verdicts. Implement escalation to human moderators for borderline cases. Use voting or consensus mechanisms across agents for reliability.

Collect Indian legal documents (court judgments, statutes, legal opinions) and create instruction pairs for tasks like summarisation, clause extraction, and case law reference. Include Indian legal terminology and citation formats. Use QLoRA on a capable base model. Validate against expert-labelled outputs. Handle multiple Indian languages. Ensure the model correctly references Indian statutes and precedents.

Collect domain-specific query-passage pairs from search logs or expert annotation. Create training triplets with hard negatives (similar but irrelevant passages). Fine-tune using contrastive learning (InfoNCE or multiple negatives ranking loss). Evaluate with retrieval metrics on a held-out test set. Use a pre-trained model like BGE or E5 as the base. Typically 5K-50K pairs suffice.

Collect Indian documents (Aadhaar, PAN cards, invoices in multiple scripts) with annotated fields. Use a pre-trained VLM like LLaVA or PaLI as the base. Create instruction data for OCR, field extraction, and document classification. Handle multiple Indian scripts (Devanagari, Tamil, Bengali, etc.). Fine-tune with LoRA on vision and language components. Evaluate on diverse document formats and quality levels.

Design multi-turn test scenarios with realistic conversation flows. Evaluate context retention (does the model remember earlier information?), coherence (are responses consistent?), and task completion over the full conversation. Measure conversation success rate, turns to resolution, and user satisfaction. Test with conversation branching and topic switches. Track per-turn and per-conversation metrics separately.

Develop task-specific test sets covering major Indian languages (Hindi, Tamil, Telugu, Bengali, etc.). Include code-mixed data (Hinglish, Tanglish). Address script variations and transliterations. Recruit native speaker evaluators for each language. Account for dialectal variations. Create culturally relevant test cases. Measure per-language performance to identify disparities. Share benchmarks with the community.