Data Engineer

Data is the foundation of AI. Models learn patterns from data, so poor quality data with noise, missing values, duplicates, or biases leads to unreliable models. The garbage-in-garbage-out principle applies strongly. Data cleaning, validation, and augmentation are critical preprocessing steps before model training.

Python lists are dynamic, can hold mixed types, and support general-purpose operations. NumPy arrays are fixed-type, contiguous in memory, and support vectorised operations that are 10-100x faster for numerical computation. Use lists for general data structures and NumPy arrays for any numerical or matrix computation in ML pipelines.

Type hints annotate function parameters and return types (e.g., def predict(input: np.ndarray) -> np.ndarray). They improve code readability, enable static analysis tools like mypy to catch bugs before runtime, and provide better IDE support. They are critical for ML production code where data type mismatches cause silent errors.

RAG combines information retrieval with LLM generation. It retrieves relevant documents from a knowledge base and includes them in the prompt context, allowing the LLM to generate answers grounded in specific data. RAG reduces hallucination, enables access to private/current data without fine-tuning, and is more cost-effective than retraining models.

A RAG pipeline consists of: (1) document ingestion and chunking, (2) embedding generation for chunks, (3) vector storage in a database, (4) query embedding and similarity search at inference time, (5) context assembly with retrieved chunks, and (6) LLM generation using the augmented prompt. Each component can be optimised independently.

A vector database stores and indexes high-dimensional vector embeddings for efficient similarity search. Unlike traditional databases that use exact matching on structured fields, vector databases use approximate nearest neighbour (ANN) algorithms like HNSW or IVF for fast similarity search. Popular options include Pinecone, Weaviate, Qdrant, ChromaDB, and pgvector.

MLOps (Machine Learning Operations) applies DevOps principles to ML systems, covering the full lifecycle from data preparation through model deployment and monitoring. It is important because most ML models fail to reach production without proper engineering practices. MLOps ensures reproducibility, reliability, scalability, and continuous improvement of ML systems in production environments.

An ML pipeline includes data ingestion and validation, feature engineering and storage, model training and hyperparameter tuning, model evaluation and validation, model registry and versioning, deployment (serving infrastructure), monitoring and alerting, and feedback loops for retraining. Each component should be automated, reproducible, and version-controlled.

Batch inference processes large volumes of data periodically (hourly, daily) and stores results for later use. Real-time inference processes individual requests with low latency (milliseconds). Batch is simpler and cheaper for recommendations or scoring; real-time is needed for fraud detection, search ranking, and conversational AI. Many systems use both: batch for pre-computation and real-time for personalisation.

Docker containers package the model, code, dependencies, and runtime into a portable, reproducible unit. This eliminates 'works on my machine' issues, ensures consistent behaviour across environments, simplifies deployment, and enables scaling via orchestrators like Kubernetes. Use multi-stage builds to minimise image size and separate training from serving containers.

GPUs (NVIDIA A100, H100) are the dominant hardware for training and inference due to their massive parallelism. TPUs (Google's custom chips) are optimised for tensor operations. CPUs handle preprocessing and lightweight inference. Specialised accelerators like AWS Inferentia target inference cost-efficiency. FPGAs offer programmable acceleration for specific workloads.

Use chunked reading with pd.read_csv(chunksize=...), specify dtypes to reduce memory, use category type for low-cardinality columns, and select only needed columns. For larger scale, consider Dask, Polars, or Vaex for out-of-core computation. Profile memory with df.info(memory_usage='deep') and optimise column types aggressively.

The Global Interpreter Lock (GIL) prevents multiple threads from executing Python bytecode simultaneously. For CPU-bound ML code, it limits true parallelism. However, most ML libraries (NumPy, PyTorch, TensorFlow) release the GIL during computation by running native C/C++/CUDA code. For true Python parallelism, use multiprocessing instead of threading.

Generators are functions that yield items one at a time using the yield keyword, maintaining state between calls. They produce items lazily, consuming minimal memory regardless of dataset size. They are essential for data loading pipelines where the entire dataset cannot fit in memory, and PyTorch DataLoader uses generators internally.

Use cProfile or line_profiler to identify bottlenecks. Replace Python loops with vectorised NumPy/pandas operations. Use Cython or Numba for CPU-intensive functions. For GPU code, use torch.cuda.Event for timing and nvidia-smi for utilisation monitoring. Memory profiling with memory_profiler helps find leaks in data pipelines.

groupby splits a DataFrame by categorical columns, and apply runs a function on each group. For feature engineering, compute group-level statistics like mean, count, or std as features. Example: df.groupby('customer_id')['amount'].transform('mean') adds per-customer average amount as a feature. Use agg for multiple statistics at once.

Multithreading shares memory but is limited by the GIL for CPU tasks, suitable for IO-bound work like data downloading. Multiprocessing creates separate processes with their own memory, bypassing the GIL for true parallelism on CPU-bound tasks. For ML data preprocessing, multiprocessing is preferred; for async API calls, threading or asyncio suffices.

Test each transformation function with known inputs and expected outputs. Test edge cases like empty DataFrames, missing values, and unexpected types. Use pytest fixtures for reusable test data. Test the full pipeline end-to-end with a small sample. Verify output shapes, dtypes, and value ranges. Use hypothesis library for property-based testing of transformations.

asyncio provides an event loop for concurrent IO-bound operations using async/await syntax. In AI applications, use it for making concurrent API calls to LLM providers, handling multiple WebSocket connections for real-time inference, or orchestrating parallel data fetches in a RAG pipeline. It does not help with CPU-bound computation.

Polars is a DataFrame library written in Rust that offers lazy evaluation, multi-threaded execution, and memory-efficient processing. It can be 10-100x faster than pandas on large datasets. Unlike pandas, it uses Apache Arrow columnar format and supports lazy query optimisation. It is increasingly adopted in production ML pipelines where pandas becomes a bottleneck.

Dataclasses provide a decorator to auto-generate __init__, __repr__, and comparison methods for data-holding classes. Pydantic models add runtime validation, serialisation, and type coercion. Use dataclasses for internal data structures like model configs. Use Pydantic for API request/response schemas and configuration files where validation of external inputs is critical.

Fixed-size chunking splits by token/character count with overlap. Semantic chunking splits at natural boundaries (paragraphs, sections). Recursive chunking tries multiple separators hierarchically. Document-structure-based chunking uses headings and sections. Choose based on document type: structured documents benefit from semantic chunking, while unstructured text works with fixed-size plus overlap.

Hybrid search combines dense vector search (semantic similarity) with sparse keyword search (BM25/TF-IDF). Dense search captures semantic meaning but can miss exact terms. Sparse search is excellent for specific keywords, names, and codes. Combining both with reciprocal rank fusion (RRF) or weighted scoring captures both semantic and lexical matches.

Implement incremental indexing that detects changed documents via checksums or timestamps. Use metadata to track document versions and update timestamps. For critical applications, maintain version history and allow querying against specific versions. Implement a document deletion pipeline that removes stale vectors. Schedule periodic full re-indexing for consistency.

Use table detection models or specialised PDF parsers (like Unstructured, Camelot, or Tabula) to extract tables. Convert tables to markdown or structured text preserving column relationships. Store tables as separate chunks with table-specific metadata. Consider embedding tables alongside their captions and surrounding context. For complex tables, create text descriptions.

Metadata filtering narrows the search space before or after vector search, improving relevance and performance. Design metadata fields based on common query patterns: date ranges for recency, categories for topic scoping, access levels for permissions, and source types for authority. Keep metadata normalised and indexed. Implement pre-filtering for mandatory constraints and post-filtering for preferences.

Expose tools like query_database(sql) with read-only access, list_tables(), and describe_table(name). Use resources to expose schema information. Implement SQL injection prevention by parameterising queries. Add result size limits and timeout handling. Restrict to SELECT queries only for safety. Include proper error messages for invalid queries. Support connection pooling for performance.

A feature store is a centralised repository for storing, managing, and serving features for ML models. It ensures consistency between training and serving features, enables feature reuse across teams, handles feature versioning, and provides both batch and real-time feature serving. Popular options include Feast, Tecton, and Hopsworks. It prevents training-serving skew.

Training-serving skew occurs when the feature computation during training differs from serving, leading to degraded model performance. Causes include different code paths, different libraries, or timing differences. Prevent it by using a feature store for consistent feature computation, sharing preprocessing code between training and serving, and implementing data validation checks.

Kubeflow is an open-source ML platform built on Kubernetes that provides components for the full ML lifecycle. It includes Kubeflow Pipelines for workflow orchestration, KFServing for model serving, Katib for hyperparameter tuning, and notebook servers for development. It enables scalable, reproducible ML workflows on any Kubernetes cluster, popular in Indian enterprises using on-premise infrastructure.

Define data schemas with expected types, ranges, and constraints using tools like Great Expectations or TFX Data Validation. Check for missing values, outliers, and distribution changes. Validate referential integrity and business rules. Run validation at data ingestion, after feature engineering, and before training. Fail the pipeline early if validation fails. Generate data quality reports.

DVC tracks large files, datasets, and ML model artifacts alongside Git without storing them in the Git repository itself. It stores metadata in Git while the actual data lives in remote storage (S3, GCS, Azure Blob). DVC enables reproducible ML experiments by versioning data alongside code, supports pipeline definition, and enables data sharing across teams.

Never hardcode secrets in code or configuration files. Use secret management services (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault). Inject secrets as environment variables at runtime. Use IAM roles and service accounts instead of API keys where possible. Rotate credentials regularly. Audit secret access. Scan repositories for accidentally committed secrets using tools like git-secrets.

Implement structured logging at each pipeline stage. Use monitoring tools (Datadog, Grafana, CloudWatch) for pipeline health dashboards. Set up alerts for failures, SLA breaches, and quality degradation. Implement retry logic with exponential backoff for transient failures. Create runbooks for common failure scenarios. Use dead letter queues for failed data records. Post-mortem significant incidents.

Cloud offers flexibility, scalability, and no upfront capital. On-premise provides lower long-term costs for sustained workloads and data sovereignty. Cloud is better for experimentation, burst capacity, and teams without infrastructure expertise. On-premise makes sense for consistent high utilisation (>70%), sensitive data requirements (Indian data localisation), or large-scale training where reserved instances cost more than owned hardware.

Define scaling metrics: GPU utilisation, request queue depth, latency percentiles, or custom business metrics. Set target values and scaling thresholds. Use Kubernetes HPA (Horizontal Pod Autoscaler) or cloud auto-scaling groups. Implement warm-up time for new instances (model loading takes time). Set minimum replicas for baseline traffic. Use predictive scaling for known traffic patterns.

Evaluate: query performance (latency at scale), scalability (data size and QPS), index types supported (HNSW, IVF), filtering capabilities (metadata filters), managed vs self-hosted, cost model, backup and replication, cloud region availability (Indian regions), API compatibility with your stack, and community/support quality. Popular choices include Pinecone, Weaviate, Qdrant, Milvus, and pgvector.

Kubernetes orchestrates containerised applications across clusters of machines. For ML, it provides GPU scheduling and sharing, auto-scaling of inference services, resource isolation for multiple teams, rolling deployments with zero downtime, self-healing when containers fail, and infrastructure-as-code for reproducibility. It is the standard platform for production ML serving.

Use object storage (S3, GCS, Azure Blob) as the foundation. Implement a medallion architecture: bronze (raw data), silver (cleaned and validated), gold (feature-ready). Use Delta Lake or Iceberg for ACID transactions and time travel. Catalog data with Apache Hive or Unity Catalog. Implement access controls and data lineage tracking. Optimise with columnar formats (Parquet) and partitioning.

Use DCGM (Data Center GPU Manager) for GPU metrics. Monitor GPU utilisation, memory usage, temperature, power consumption, and ECC errors. Integrate with Prometheus for collection and Grafana for visualisation. Set alerts for low utilisation (wasted resources), high temperature (throttling risk), and ECC errors (hardware failure). Track GPU time per team for chargeback. Monitor driver and CUDA version consistency.

Use a combination of rule-based filters for known fraud patterns and ML models (gradient boosting or deep learning) for anomaly detection. Feature engineer from transaction metadata, user behavior, device info, and network graphs. Handle extreme class imbalance with SMOTE or focal loss. Deploy with real-time inference under 100ms latency and continuous model retraining.

Use Apache Kafka or Redis Streams for message ingestion, asyncio or Faust for stream processing, and a FastAPI/gRPC endpoint for inference. Implement windowed feature computation with a sliding buffer. Use connection pooling for database lookups and batch inference requests when possible. Monitor latency and throughput with Prometheus metrics.

Ingest PDFs of policy documents with structure-aware parsing for Indian government formatting. Handle Hindi and English content with multilingual embeddings. Implement metadata filtering by ministry, date, and policy type. Use hybrid search for exact policy numbers and semantic queries. Include citation generation with specific section references. Handle amendments and superseded policies.

Store access control metadata (tenant, department, role) with each chunk. Apply metadata filters at query time to enforce permissions. Integrate with the enterprise's existing IAM (like Active Directory). Ensure vector database supports metadata filtering efficiently. Implement audit logging for compliance. Handle Indian data localisation requirements by keeping sensitive data in India-based storage.

Cache frequent queries and their results. Use approximate nearest neighbour indices for faster search. Implement query embedding caching. Batch vector lookups. Use streaming for LLM generation. Pre-compute and store embeddings for static content. Optimise chunk sizes to reduce the number of retrievals. Consider using smaller, faster models for retrieval stages.

Expose tools for checking transaction status, viewing payment history, and generating payment links. Use resources for account balance and recent transactions. Implement strict authentication with OAuth 2.0 and require human approval for any payment-related actions. Handle UPI-specific formats (VPA validation, transaction IDs). Comply with RBI guidelines for data handling and ensure audit logging.

Create separate MCP servers for each domain: HR (employee data, leave management), Finance (invoicing, expense reports), CRM (customer data), and IT (ticket management). Implement a gateway server for authentication and routing. Use SSE transport for shared remote servers. Standardise tool naming and error handling across servers. Document tools for both LLMs and developers.

Use a model serving framework (TorchServe, Triton, or TF Serving) behind a load balancer. Deploy in Indian cloud regions (AWS Mumbai, Azure Central India) for low latency. Implement auto-scaling based on traffic patterns (handle sale events like Diwali/Republic Day). Use model caching and batching for efficiency. Set up health checks, monitoring, and automatic failover. Consider edge deployment for latency-critical features.

Start with managed services to move fast: GitHub for code, DVC for data versioning, MLflow for experiment tracking, AWS SageMaker or GCP Vertex AI for training and serving. Implement CI/CD with GitHub Actions. Use Docker and Kubernetes for serving. Set up monitoring with Prometheus and Grafana. Automate data validation and model evaluation. Plan for compliance with Indian data regulations. Scale infrastructure as the team grows.

Use a streaming platform (Kafka, Kinesis) for event ingestion. Process events with a stream processor (Flink, Spark Streaming) for feature computation. Store computed features in a low-latency store (Redis, DynamoDB) for serving. Implement exactly-once processing semantics. Handle late-arriving data with watermarks. Monitor pipeline lag and throughput. Ensure consistency with batch-computed features.

Replicate model artifacts and configurations across regions. Use multi-region deployment with failover routing. Implement automated health checks and failover triggers. Back up model registries and feature stores. Document recovery procedures and test them regularly. Define RTO (Recovery Time Objective) and RPO (Recovery Point Objective). Keep model serving stateless for easy recovery.

Use distributed file systems (HDFS, Lustre) or object storage (S3) for training data. Implement data sharding across storage nodes. Use streaming data loaders that prefetch and pipeline IO with computation. Cache frequently accessed data on local NVMe SSDs. Optimise data format (TFRecord, WebDataset, Parquet) for sequential read performance. Monitor IO bottlenecks with storage metrics.

Deploy Triton or a custom serving layer on Kubernetes with GPU node pools in Indian cloud regions. Implement model routing based on request type. Use shared GPU scheduling to maximise utilisation across models. Implement a model registry for version management. Set up centralised monitoring and cost attribution per team. Handle compliance with Indian data regulations. Implement rate limiting and authentication per consumer.