Data Scientist

AI refers to systems that can perform tasks typically requiring human intelligence, such as learning, reasoning, and perception. Unlike traditional software that follows explicit rules, AI systems learn patterns from data and can generalise to new inputs.

AI is the broadest term covering any technique that enables machines to mimic human intelligence. ML is a subset of AI where models learn from data without explicit programming. Deep Learning is a subset of ML using neural networks with multiple layers to learn hierarchical representations.

Supervised learning uses labelled data (e.g., spam detection). Unsupervised learning finds patterns in unlabelled data (e.g., customer segmentation for Indian e-commerce platforms). Reinforcement learning learns via rewards (e.g., training a robot to navigate a warehouse).

A neural network is a computational model composed of layers of interconnected nodes (neurons) that process inputs through weighted connections. It is inspired by biological neurons but the analogy is loose. Each artificial neuron applies a weighted sum followed by a non-linear activation function.

Indian companies use AI extensively in fintech (credit scoring for underbanked populations), agriculture (crop disease detection), healthcare (radiology screening at scale), e-commerce (recommendation engines and vernacular search), and customer support (multilingual chatbots supporting Indian languages).

AI is like teaching a computer to learn from examples rather than following fixed rules. For instance, instead of writing rules to detect fraudulent UPI transactions, we show the system millions of past transactions labelled as fraud or legitimate, and it learns the patterns itself. It gets better with more data and can handle cases we never explicitly programmed.

Data is the foundation of AI. Models learn patterns from data, so poor quality data with noise, missing values, duplicates, or biases leads to unreliable models. The garbage-in-garbage-out principle applies strongly. Data cleaning, validation, and augmentation are critical preprocessing steps before model training.

Classification predicts discrete categorical labels (e.g., spam or not spam, disease type), while regression predicts continuous numerical values (e.g., house price, temperature). Classification uses metrics like accuracy, precision, and recall, while regression uses MSE, MAE, and R-squared.

Python offers a rich ecosystem of ML libraries (NumPy, pandas, scikit-learn, PyTorch, TensorFlow), clean syntax that enables rapid prototyping, strong community support, and excellent interoperability with C/C++ for performance-critical components. Its interpreted nature allows interactive development through Jupyter notebooks, which is ideal for experimentation.

Python lists are dynamic, can hold mixed types, and support general-purpose operations. NumPy arrays are fixed-type, contiguous in memory, and support vectorised operations that are 10-100x faster for numerical computation. Use lists for general data structures and NumPy arrays for any numerical or matrix computation in ML pipelines.

Use NumPy: def cosine_similarity(a, b): return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)). This computes the dot product of the vectors divided by the product of their magnitudes. The result ranges from -1 (opposite) to 1 (identical direction). Add epsilon to the denominator to prevent division by zero.

Use virtual environments (venv, conda, or poetry) to isolate project dependencies. Pin exact versions in requirements.txt or pyproject.toml for reproducibility. Use conda for complex native dependencies like CUDA. Docker containers ensure full environment reproducibility. Tools like pip-tools or poetry provide dependency resolution and lock files.

First understand the missing data pattern (MCAR, MAR, MNAR) using df.isnull().sum() and missingno library. Options include dropping rows/columns with too many missing values, imputing with mean/median/mode, using advanced imputation like KNN or iterative imputer, or using algorithms that handle NaN natively like XGBoost. Document your strategy for reproducibility.

A scalar is a single number (0D), a vector is a 1D array of numbers, a matrix is a 2D array, and a tensor is a generalisation to any number of dimensions. Each is a special case of a tensor. In ML, model weights are stored as tensors, and frameworks like PyTorch and NumPy operate on these n-dimensional arrays.

The dot product of two vectors is the sum of their element-wise products, producing a scalar. It measures similarity between vectors. In neural networks, each neuron computes a dot product between its weights and inputs, followed by a bias and activation function. Matrix multiplication, which powers layer computations, is built from dot products.

Gradient descent is an iterative optimisation algorithm that updates parameters in the direction opposite to the gradient of the loss function. The gradient points in the direction of steepest ascent, so moving against it reduces the loss. The learning rate controls step size. Variants include SGD, Adam, and AdaGrad, each handling learning rates differently.

Softmax converts a vector of real numbers (logits) into a probability distribution by exponentiating each element and normalising by the sum: softmax(z_i) = exp(z_i) / sum(exp(z_j)). It ensures outputs are positive and sum to 1, making them interpretable as class probabilities. It is used in the final layer of multi-class classification networks.

A probability distribution describes how probabilities are assigned to different outcomes. The Gaussian (normal) distribution is bell-shaped, defined by mean and variance, and arises naturally due to the Central Limit Theorem. The Bernoulli distribution models binary outcomes with probability p for success. These are foundational in ML for modelling data and defining loss functions.

Correlation means two variables tend to change together but does not imply one causes the other. Causation requires a direct mechanism. ML models primarily learn correlations, which can lead to spurious predictions when the underlying causal relationship changes. Understanding this distinction is critical for model robustness, fairness, and deployment in high-stakes domains.

Cosine similarity measures the angle between vectors (direction), ranging from -1 to 1. Euclidean distance measures the straight-line distance between points (magnitude matters). Use cosine similarity when only direction matters (text embeddings, recommendation systems). Use Euclidean distance when magnitude is meaningful (spatial data, anomaly detection based on deviation).

A deterministic model always produces the same output for the same input. A stochastic model incorporates randomness and may produce different outputs each time. Most ML training is stochastic (random initialisation, mini-batch sampling, dropout). Stochasticity often helps by preventing overfitting and escaping local minima during optimisation.

Accuracy is the fraction of all predictions that are correct. Precision is the fraction of positive predictions that are correct (low false positive rate). Recall is the fraction of actual positives that are correctly identified (low false negative rate). F1 is the harmonic mean of precision and recall, useful when both matter equally. Use the metric that aligns with the cost of errors in your application.

Cross-validation splits data into k folds, trains on k-1 folds and evaluates on the remaining fold, rotating through all folds. It provides a more reliable performance estimate than a single train-test split, especially with limited data. Use k-fold (typically k=5 or 10) for model selection, stratified k-fold for imbalanced classes, and leave-one-out for very small datasets.

Experiment tracking records hyperparameters, metrics, code versions, data versions, and artifacts for each training run. It enables comparison across experiments and reproducibility. Popular tools include MLflow (open-source, self-hosted), Weights & Biases (cloud-based with rich visualisation), Neptune, and Comet. Choose based on team size, budget, and integration requirements.

AI bias occurs when a model produces systematically unfair outcomes due to biased training data, flawed feature selection, or skewed labelling. This matters because biased models can discriminate against underrepresented groups, especially critical in applications like loan approvals, hiring, and criminal justice.

Parametric models have a fixed number of parameters regardless of data size (e.g., linear regression, logistic regression). Non-parametric models grow in complexity with more data (e.g., k-NN, decision trees). Parametric models are faster but may underfit; non-parametric models are more flexible but can overfit.

Overfitting occurs when a model memorises training data and performs poorly on unseen data, detected by a large gap between training and validation accuracy. Underfitting means the model is too simple to capture patterns, shown by poor performance on both sets. Regularisation, dropout, and cross-validation address overfitting; increasing model complexity or features addresses underfitting.

The bias-variance tradeoff describes the tension between a model's ability to fit training data (low bias) and its ability to generalise to new data (low variance). High bias leads to underfitting while high variance leads to overfitting. The goal is to find the optimal complexity that minimises total error.

Start with exploratory data analysis and feature importance analysis. Reduce dimensionality using PCA or feature selection methods. Split data into train/validation/test sets. Begin with a simple baseline like logistic regression, then try gradient boosting (XGBoost/LightGBM) for structured data. Use cross-validation for hyperparameter tuning and evaluate with appropriate metrics.

Discriminative models learn the decision boundary between classes by modelling P(y|x) directly, such as logistic regression and SVMs. Generative models learn the joint probability P(x,y) or P(x|y) to understand how data is generated, such as Naive Bayes and GANs. Generative models can create new data samples while discriminative models typically cannot.

Embeddings are dense, low-dimensional vector representations of high-dimensional data like words, images, or users. They capture semantic relationships so that similar items have nearby vectors. They enable efficient similarity search, are used in recommendation systems, and form the foundation of modern NLP models.

Batch gradient descent computes gradients over the entire dataset per update, which is stable but slow. Stochastic gradient descent (SGD) updates weights after each sample, which is noisy but fast. Mini-batch gradient descent is the practical middle ground, computing gradients over small batches (32-512 samples), offering both speed and stability.

Key considerations include ensuring fairness across diverse Indian demographics (caste, language, region), respecting data privacy under India's DPDP Act, maintaining transparency in automated decision-making, preventing displacement of workers without reskilling programs, and ensuring AI systems work equitably for rural and urban populations.

Precision is the fraction of positive predictions that are correct; recall is the fraction of actual positives correctly identified. F1 score is their harmonic mean. Prioritise recall in medical diagnosis (missing a disease is costly), precision in spam filtering (false positives are annoying), and F1 when both matter equally.

Regularisation adds a penalty term to the loss function to prevent overfitting. L1 (Lasso) adds the absolute value of weights, encouraging sparsity and effective feature selection. L2 (Ridge) adds the squared weights, shrinking all weights toward zero without eliminating them. Elastic Net combines both approaches.

AutoML automates the process of selecting models, tuning hyperparameters, and engineering features. Tools like Google AutoML, H2O, and Auto-sklearn make ML accessible to non-experts and accelerate prototyping. However, domain expertise and understanding of results remain essential for production deployments.

Broadcasting is NumPy's mechanism for performing operations on arrays of different shapes. It automatically expands the smaller array to match the larger one without copying data. Rules require dimensions to be either equal or one of them to be 1, aligning from the trailing dimension. This enables concise and efficient vectorised code.

Use chunked reading with pd.read_csv(chunksize=...), specify dtypes to reduce memory, use category type for low-cardinality columns, and select only needed columns. For larger scale, consider Dask, Polars, or Vaex for out-of-core computation. Profile memory with df.info(memory_usage='deep') and optimise column types aggressively.

Decorators are functions that modify the behaviour of other functions, applied using the @decorator syntax. In ML codebases, they are used for logging (tracking experiments), timing (measuring function performance), caching (memoising expensive computations), and in frameworks like PyTorch for torch.no_grad() and custom autograd functions.

Generators are functions that yield items one at a time using the yield keyword, maintaining state between calls. They produce items lazily, consuming minimal memory regardless of dataset size. They are essential for data loading pipelines where the entire dataset cannot fit in memory, and PyTorch DataLoader uses generators internally.

Shallow copy creates a new object but references the same nested objects; deep copy recursively copies everything. In ML, this matters when modifying model configurations or data preprocessing pipelines. Accidentally modifying shared references can corrupt data or cause subtle training bugs. Use copy.deepcopy() for model state dicts and configuration objects.

groupby splits a DataFrame by categorical columns, and apply runs a function on each group. For feature engineering, compute group-level statistics like mean, count, or std as features. Example: df.groupby('customer_id')['amount'].transform('mean') adds per-customer average amount as a feature. Use agg for multiple statistics at once.

Polars is a DataFrame library written in Rust that offers lazy evaluation, multi-threaded execution, and memory-efficient processing. It can be 10-100x faster than pandas on large datasets. Unlike pandas, it uses Apache Arrow columnar format and supports lazy query optimisation. It is increasingly adopted in production ML pipelines where pandas becomes a bottleneck.

Pickle serialises Python objects to byte streams for storage and loading. It is commonly used to save scikit-learn models and data. However, pickle can execute arbitrary code during deserialisation, making it a security risk for untrusted sources. Use safer alternatives like ONNX, safetensors, or joblib with known-source files. Never unpickle data from untrusted origins.

Sklearn Pipelines chain preprocessing and model steps so that transformations are fit only on training data and applied consistently to test data. Use Pipeline with ColumnTransformer for feature-specific preprocessing. This prevents leakage from fitting scalers or encoders on the full dataset. Always call pipeline.fit(X_train) and pipeline.predict(X_test) rather than transforming separately.

Context managers implement __enter__ and __exit__ methods and are used with the 'with' statement for resource management. In ML, they are used for managing GPU memory (torch.no_grad(), torch.cuda.amp.autocast()), file handles for large data loading, database connections, and experiment tracking contexts like mlflow.start_run().

Eigenvectors of a matrix are vectors whose direction remains unchanged when multiplied by that matrix; eigenvalues are the scaling factors. In ML, they are central to PCA for dimensionality reduction, spectral clustering, understanding covariance structures, and analysing the stability of dynamical systems and recurrent networks.

Bayes' theorem states P(A|B) = P(B|A) * P(A) / P(B), relating posterior probability to likelihood, prior, and evidence. In ML, Naive Bayes classifiers use it for text classification. Bayesian inference updates beliefs about model parameters as new data arrives. It is also fundamental to probabilistic graphical models and Bayesian neural networks.

PCA finds the directions (principal components) of maximum variance in the data by computing the eigenvectors of the covariance matrix. Data is projected onto the top-k eigenvectors to reduce dimensionality while preserving the most variance. Mathematically, it is equivalent to SVD of the centered data matrix. It assumes linear relationships between features.

The L1 norm is the sum of absolute values of vector elements, while the L2 norm is the square root of the sum of squared elements. L1 regularisation (Lasso) promotes sparsity by driving weights to exactly zero, useful for feature selection. L2 regularisation (Ridge) distributes penalty evenly, shrinking all weights. L1 has diamond-shaped constraint regions; L2 has circular ones.

Cross-entropy loss measures the difference between predicted probability distribution and true distribution: -sum(y_i * log(p_i)). It is preferred over MSE for classification because it penalises confident wrong predictions heavily, has better gradient properties (avoids saturation with sigmoid/softmax), and is derived from maximum likelihood estimation for categorical distributions.

The CLT states that the mean of many independent random variables tends toward a normal distribution, regardless of the original distribution. In ML, this justifies using Gaussian assumptions in many models, underpins statistical testing of model performance, explains why batch normalisation works, and is why many natural phenomena are approximately normally distributed.

A loss function quantifies the discrepancy between model predictions and true values. For regression, use MSE (sensitive to outliers) or MAE (robust). For classification, use cross-entropy. For ranking, use hinge loss or triplet loss. For imbalanced data, use focal loss or weighted cross-entropy. The choice depends on the task, data distribution, and what errors are most costly.

As dimensions increase, data becomes increasingly sparse, and distance metrics become less meaningful because points tend to be equidistant. This makes nearest-neighbour methods ineffective, increases the data needed for reliable estimation exponentially, and complicates optimisation. Dimensionality reduction techniques like PCA, t-SNE, or autoencoders help mitigate this problem.

Entropy measures the uncertainty or randomness in a probability distribution: H = -sum(p_i * log(p_i)). Maximum entropy means maximum uncertainty (uniform distribution). In decision trees, information gain (reduction in entropy) is used to select the best feature to split on at each node, choosing the feature that most reduces uncertainty about the target variable.

Create instruction-response pairs in the model's expected format. Ensure diversity in instructions, response lengths, and complexity. Include edge cases and negative examples. Clean data for quality: remove duplicates, fix formatting, validate factual accuracy. Aim for 1K-100K high-quality examples. Use consistent formatting with clear instruction, input, and output fields.

High-quality data consistently outperforms large quantities of noisy data for fine-tuning. A few thousand expert-curated examples often beat tens of thousands of auto-generated ones. Focus on diverse, representative, and accurately labelled examples. Clean data aggressively. However, some quantity is needed for generalisation. The sweet spot is typically 1K-10K high-quality examples for most tasks.

Use a stronger model (like GPT-4 or Claude) to generate instruction-response pairs. Provide diverse seed prompts covering your domain. Apply quality filters to remove low-quality generations. Validate with domain experts for a subset. Augment with paraphrasing and variation generation. Be careful about model distillation terms of service. Combine synthetic data with real examples for best results.

Collect diverse, representative examples from real usage patterns. Include edge cases, adversarial inputs, and failure modes. Have domain experts create ground truth answers. Ensure coverage across different input types, lengths, and topics. Version the test set and keep it separate from training data. Update periodically to reflect evolving usage. Aim for 100-1000 examples depending on task complexity.

A/B testing compares two model versions with real users by randomly assigning traffic. Design: define a clear hypothesis and primary metric, calculate required sample size for statistical power, randomise at the user level to avoid cross-contamination, run for sufficient duration to capture temporal patterns, and use proper statistical tests (not just point estimates) for decision making.

Data drift occurs when the statistical properties of production data change from training data distribution. Detect it using statistical tests (KS test, PSI) on feature distributions, monitoring prediction confidence distributions, and tracking feature statistics over time. Handle it by triggering retraining pipelines, alerting data teams, and implementing adaptive models. Monitor continuously, not just at deployment.

Use a traffic splitting layer to route requests to different model versions. Randomise at the user level for consistency. Define clear success metrics and minimum sample sizes before starting. Run for sufficient duration to capture weekly patterns. Use proper statistical tests and correct for multiple comparisons. Implement guardrail metrics that trigger automatic rollback if violated.

Use a combination of rule-based filters for known fraud patterns and ML models (gradient boosting or deep learning) for anomaly detection. Feature engineer from transaction metadata, user behavior, device info, and network graphs. Handle extreme class imbalance with SMOTE or focal loss. Deploy with real-time inference under 100ms latency and continuous model retraining.

Interpretability means a model's internal logic is inherently understandable (e.g., decision trees), while explainability uses post-hoc methods (SHAP, LIME) to approximate why a complex model made a decision. In regulated Indian industries like banking (RBI guidelines) and insurance (IRDAI), explainability is often mandated for automated decisions affecting customers.

SVD decomposes any matrix A into three matrices: A = U * Sigma * V^T, where U and V are orthogonal and Sigma is diagonal with singular values. In ML, it is used for PCA, matrix factorisation in recommendation systems, low-rank approximation for model compression, and latent semantic analysis in NLP.

Give the agent tools for data loading (CSV, SQL, APIs), Python code execution for analysis, chart generation, and report writing. Implement a sandboxed code execution environment. Include data profiling as a first step. Handle large datasets with sampling strategies. Add validation tools to verify statistical conclusions. Include guardrails against executing harmful code.