Evaluating AI systems

Evals are systematic methods for measuring AI system performance across quality, safety, and reliability dimensions. They are critical because they prevent deploying underperforming or harmful models, enable data-driven decision making between model versions, catch regressions early, and build stakeholder confidence. Without evals, AI development is essentially flying blind.

Offline evaluation tests models on static datasets before deployment using metrics like accuracy, F1, and perplexity. Online evaluation measures model performance in production with real users through A/B testing, user satisfaction, and business metrics. Offline catches obvious issues early; online captures real-world performance that test sets cannot fully represent. Both are essential.

Accuracy is the fraction of all predictions that are correct. Precision is the fraction of positive predictions that are correct (low false positive rate). Recall is the fraction of actual positives that are correctly identified (low false negative rate). F1 is the harmonic mean of precision and recall, useful when both matter equally. Use the metric that aligns with the cost of errors in your application.

Cross-validation splits data into k folds, trains on k-1 folds and evaluates on the remaining fold, rotating through all folds. It provides a more reliable performance estimate than a single train-test split, especially with limited data. Use k-fold (typically k=5 or 10) for model selection, stratified k-fold for imbalanced classes, and leave-one-out for very small datasets.

AI bias occurs when a model produces systematically unfair outcomes due to biased training data, flawed feature selection, or skewed labelling. This matters because biased models can discriminate against underrepresented groups, especially critical in applications like loan approvals, hiring, and criminal justice.

Precision is the fraction of positive predictions that are correct; recall is the fraction of actual positives correctly identified. F1 score is their harmonic mean. Prioritise recall in medical diagnosis (missing a disease is costly), precision in spam filtering (false positives are annoying), and F1 when both matter equally.

Evaluate on task-specific benchmarks with your actual data, not just public leaderboards. Compare latency, throughput, cost per token, context window size, and output quality. Consider factors like API reliability, data privacy policies, and hosting options (cloud vs self-hosted). Run A/B tests and collect human evaluation scores on representative queries.

Cross-entropy loss measures the average negative log probability of correct tokens. Perplexity is the exponential of cross-entropy and represents the effective number of equally probable token choices the model considers. Lower perplexity indicates better prediction. Perplexity is more interpretable but both metrics are mathematically equivalent for comparison.

Key risks include generating harmful content, producing convincing misinformation, leaking private training data, being manipulated through prompt injection, encoding and amplifying biases, and enabling malicious uses like phishing. Mitigations include RLHF alignment, content filtering, red-teaming, and responsible deployment practices.

Create a diverse test set of inputs with expected outputs. Measure quantitative metrics (accuracy, relevance scores, format compliance). Use LLM-as-judge for qualitative evaluation. Track prompt versions systematically. A/B test in production. Iterate by analysing failure cases and refining instructions. Tools like promptfoo, LangSmith, and Braintrust help automate this process.

Evaluate retrieval quality using recall@k, precision@k, and MRR (Mean Reciprocal Rank). Evaluate generation quality using faithfulness (is the answer grounded in retrieved context?), relevancy (does it answer the question?), and correctness. Use frameworks like RAGAS for automated evaluation. Also measure end-to-end latency and user satisfaction.

Measure task completion rate, step efficiency (number of steps to complete a task), tool usage accuracy, cost per task, latency, and error recovery success. Create benchmark tasks with known solutions. Track trajectories to understand reasoning quality. Use human evaluation for subjective quality. Compare against non-agent baselines. Monitor for regressions when changing models or prompts.

Compare against the base model on your task-specific evaluation set using relevant metrics. Check for regression on general capabilities with standard benchmarks. Compute perplexity on a held-out set. Use human evaluation for subjective quality. A/B test in production. Verify the model handles edge cases in your domain. Monitor both task performance and safety metrics.

Create a diverse test set covering common cases, edge cases, and adversarial inputs. Define metrics for each quality dimension: correctness, relevance, completeness, formatting, latency, and safety. Use automated metrics where possible, LLM-as-judge for subjective qualities, and human evaluation for ground truth. Track metrics across model versions and prompt changes. Automate evaluation in CI/CD.

LLM-as-judge uses a powerful LLM to evaluate the quality of another model's outputs by scoring or comparing responses. It scales better than human evaluation and correlates well with human judgments for many criteria. Limitations include self-preference bias, inconsistency across runs, sensitivity to evaluation prompt phrasing, and inability to catch factual errors without ground truth.

For retrieval: precision@k, recall@k, MRR, and NDCG to measure document relevance. For generation: faithfulness (is the answer grounded in context?), answer relevance (does it address the question?), and completeness. End-to-end: accuracy against ground truth, hallucination rate, latency, and cost per query. Use frameworks like RAGAS or custom evaluation scripts.

Collect diverse, representative examples from real usage patterns. Include edge cases, adversarial inputs, and failure modes. Have domain experts create ground truth answers. Ensure coverage across different input types, lengths, and topics. Version the test set and keep it separate from training data. Update periodically to reflect evolving usage. Aim for 100-1000 examples depending on task complexity.

Intrinsic evaluation measures the model's core capabilities in isolation using task-specific benchmarks (perplexity, BLEU, accuracy). Extrinsic evaluation measures how the model performs in the end application, affecting real-world outcomes (user satisfaction, task completion rate, business metrics). A model with better intrinsic scores may not always lead to better extrinsic performance.

AI regression testing verifies that model updates do not degrade performance on previously working cases. Unlike software regression testing with deterministic pass/fail, AI regression testing deals with probabilistic outputs and requires statistical comparison. Key differences include tolerance thresholds for quality changes, the need for representative test distributions, and the possibility of acceptable trade-offs between metrics.

Compare against known ground truth databases or verified sources. Use claim decomposition to break responses into individual facts and verify each. Implement automated fact-checking pipelines using knowledge bases. Use NLI (natural language inference) models to check consistency between claims and source documents. Track hallucination rate as a key metric. Human spot-checking remains essential for novel claims.

A/B testing compares two model versions with real users by randomly assigning traffic. Design: define a clear hypothesis and primary metric, calculate required sample size for statistical power, randomise at the user level to avoid cross-contamination, run for sufficient duration to capture temporal patterns, and use proper statistical tests (not just point estimates) for decision making.

MMLU (Massive Multitask Language Understanding) evaluates LLMs across 57 subjects from STEM, humanities, and social sciences at varying difficulty levels. It measures a model's breadth of knowledge and reasoning through multiple-choice questions. While widely used for model comparison, it has limitations including potential training data contamination and not measuring generation quality.

Pointwise evaluation scores each response independently on a scale (e.g., 1-5 for quality). Pairwise evaluation compares two responses and selects the better one. Pairwise is easier for human evaluators and LLM judges (comparing is easier than absolute scoring) but requires more evaluations to rank many models. Pairwise is preferred for model comparison; pointwise for tracking absolute quality.

Red teaming involves deliberately trying to make AI systems fail or produce harmful outputs through creative adversarial inputs. This includes prompt injection attempts, jailbreaking, edge case exploitation, and bias probing. It uncovers vulnerabilities that standard testing misses. Effective red teaming requires diverse evaluators with different backgrounds and perspectives to cover a wide attack surface.

Data contamination occurs when evaluation benchmark data appears in a model's training data, inflating performance scores without reflecting true capability. It is a major problem because contaminated benchmarks give misleading comparisons. Detect it by checking for memorisation of test examples, using held-out data created after training cutoff, and implementing novel evaluation tasks.

Measure P50, P95, and P99 latency at each pipeline stage (preprocessing, model inference, postprocessing). Profile with tools like NVIDIA Nsight for GPU utilisation. Optimise by reducing model size (quantisation, distillation), batching requests, using faster hardware, implementing caching, and optimising tokenisation. Set latency SLAs based on user experience requirements.

BLEU only measures n-gram overlap and misses semantic quality. Better approaches include BERTScore (embedding similarity), METEOR (semantic matching with synonyms), human evaluation along dimensions like fluency, coherence, and relevance, LLM-as-judge scoring, and task-specific metrics. For summarisation, use ROUGE and factual consistency checks. Combine multiple metrics for a comprehensive view.

An evaluation rubric defines criteria and scoring scales for assessing output quality. For LLM outputs, define dimensions like correctness, completeness, clarity, formatting, and safety. Create clear scoring guidelines with examples for each level (e.g., 1=incorrect, 3=partially correct, 5=fully correct). Use the rubric consistently across human evaluators and as guidance for LLM-as-judge prompts.

Run evaluations multiple times and report mean and variance. Use temperature 0 for reproducible baseline comparisons. For sampling-based evaluation, use enough samples for statistical significance. Set random seeds where possible. Use robust metrics that tolerate minor variations. Compare distributions of scores rather than individual outputs. Report confidence intervals alongside point estimates.

Evaluate task completion rate (does it achieve the goal?), step efficiency (how many steps?), tool usage accuracy (correct tool selection and parameters), cost (total tokens consumed), error recovery (can it handle failures?), safety (no harmful actions), and reasoning quality (is the thinking sound?). Also measure user experience metrics like response time and transparency of reasoning.

The Elo rating system, adapted from chess, assigns models a rating based on head-to-head comparisons. Chatbot Arena shows users two anonymous model responses and collects preferences. Models gain or lose rating points based on wins and losses against opponents of known strength. This crowdsourced approach provides robust rankings that account for human preferences and reduces individual evaluator bias.

A model card documents the model's intended use, training data, evaluation results across subgroups, limitations, and ethical considerations. Include performance metrics disaggregated by demographic groups. Document known failure modes and biases. Specify the evaluation methodology and datasets used. Include environmental impact of training. Follow templates from Google or Hugging Face for consistency.

Development evaluation uses curated test sets to make go/no-go decisions before deployment, focusing on model quality and safety. Production monitoring tracks live performance metrics, user feedback, latency, error rates, and data drift continuously. Production monitoring catches issues that test sets miss, like distribution shifts, real-world edge cases, and gradual performance degradation over time.

Use a traffic splitting layer to route requests to different model versions. Randomise at the user level for consistency. Define clear success metrics and minimum sample sizes before starting. Run for sufficient duration to capture weekly patterns. Use proper statistical tests and correct for multiple comparisons. Implement guardrail metrics that trigger automatic rollback if violated.

Interpretability means a model's internal logic is inherently understandable (e.g., decision trees), while explainability uses post-hoc methods (SHAP, LIME) to approximate why a complex model made a decision. In regulated Indian industries like banking (RBI guidelines) and insurance (IRDAI), explainability is often mandated for automated decisions affecting customers.

Route a percentage of traffic to each prompt variant. Define clear metrics (accuracy, user satisfaction, latency, cost). Ensure sufficient sample size for statistical significance. Log all inputs, outputs, and metadata. Use automated evaluation with LLM judges plus manual review of edge cases. Implement gradual rollout with rollback capability. Monitor for regression over time.

Define a standardised task suite with varying complexity. Measure success rate, average steps to completion, cost, latency, and error recovery effectiveness. Test on edge cases and adversarial inputs. Compare single-agent vs multi-agent, different LLM backends, and various prompting strategies. Use frameworks like AgentBench or custom evaluation harnesses. Run multiple trials for statistical significance.

Test with adversarial prompts designed to elicit harmful outputs. Evaluate fairness across demographic groups using disaggregated metrics. Test for common biases (gender, caste, regional in Indian context). Use red-teaming with diverse evaluators. Check for PII leakage. Evaluate content safety with toxicity classifiers. Test prompt injection resistance. Document findings in a model card.

Integrate eval scripts into the CI pipeline triggered by model or prompt changes. Store test sets and expected outputs in version control. Run automated metrics computation and LLM-as-judge evaluations. Set quality gates with minimum threshold scores. Generate evaluation reports comparing against baselines. Alert on regressions. Use tools like promptfoo, Braintrust, or custom evaluation frameworks.

Design multi-turn test scenarios with realistic conversation flows. Evaluate context retention (does the model remember earlier information?), coherence (are responses consistent?), and task completion over the full conversation. Measure conversation success rate, turns to resolution, and user satisfaction. Test with conversation branching and topic switches. Track per-turn and per-conversation metrics separately.

Create test cases with coding problems at various difficulty levels. Measure functional correctness (do generated solutions pass test cases?), code quality (readability, efficiency), and security (no vulnerabilities). Use pass@k metric for solution accuracy. Test across multiple languages and frameworks. Evaluate explanation quality alongside code. Benchmark against existing solutions and human performance.

Develop task-specific test sets covering major Indian languages (Hindi, Tamil, Telugu, Bengali, etc.). Include code-mixed data (Hinglish, Tanglish). Address script variations and transliterations. Recruit native speaker evaluators for each language. Account for dialectal variations. Create culturally relevant test cases. Measure per-language performance to identify disparities. Share benchmarks with the community.

Create a test suite of known injection attacks: direct instruction override, indirect injection through context, encoding-based attacks, and multi-step manipulation. Measure the rate at which the model follows injected instructions versus maintaining its original behaviour. Test regularly as new attack patterns emerge. Implement continuous monitoring in production for unusual output patterns.