Data Engineer - AWS & Azure Data Platforms

DURGA R

Data Engineer Email: *******************@*****.*** Mobile: +1-512-***-**** USA

PROFESSIONAL SUMMARY

● 5+ years of experience as a Data Engineer specializing in designing and building large scale cloud data platforms across AWS and Azure ecosystems, delivering enterprise data solutions for healthcare, insurance, and banking domains.

● Built and optimized distributed data processing pipelines handling 250 million+ daily records and multi-terabyte daily ingestion using Spark, Databricks, AWS Glue, and Azure Data Factory with consistent SLA adherence above 99.5 percent.

● Designed and implemented scalable data lake and data warehouse architectures using S3, ADLS Gen2, Redshift, and Synapse, improving query performance by up to 60 percent through partitioning, modeling, and execution tuning strategies.

● Strong expertise in batch and streaming architectures using Kafka, Event Hubs, AWS Kinesis, and Spark Structured Streaming, achieving near real time processing latencies between 5 to 90 seconds depending on use case criticality.

● Experienced in end-to-end data engineering lifecycle including schema design, ETL optimization, data modeling, CI CD pipelines, and observability frameworks using CloudWatch, Azure Monitor, and automated recovery mechanisms.

TECHNICAL SKILLS

Core Area Skills / Technologies

Programming &

Scripting

Python, SQL, Shell scripting, PySpark

Data Engineering & ETL ETL/ELT pipelines, batch processing, real time streaming, data ingestion frameworks, incremental processing, CDC pipelines

Big Data Processing Apache Spark, Spark Structured Streaming, Hadoop, Hive, MapReduce, Kafka, AWS Kinesis, Azure Event Hubs

Cloud Platforms (AWS) S3, Glue, Athena, Redshift, Lambda, Step Functions, EMR, EC2, VPC, CloudWatch, Lake Formation, OpenSearch, Code Pipeline, ECS, EKS, SageMaker

Cloud Platforms (Azure) Data Factory, Data Lake Storage Gen2, Synapse Analytics, Azure SQL, Azure Functions, Event Hubs, Azure VM, Azure VNet, Azure Monitor, Azure Key Vault, Azure Databricks, Azure Blob Storage, AKS

Data Warehousing Snowflake, Amazon Redshift, Azure Synapse Analytics, SQL Server, star schema, snowflake schema, dimensional modeling, OLAP optimization Data Modeling & Schema

Design

Schema design, data modeling, normalization, denormalization, SCD Type 1/2, schema registry, data contracts, metadata driven design Orchestration & CI/CD Apache Airflow, AWS Step Functions, Azure Data Factory pipelines, Jenkins, Git, CI/CD pipelines, Docker, containerized deployments Data Visualization & BI Tableau, Power BI, Looker, dashboard optimization, KPI modeling Advanced & AI/ML

Systems

Machine Learning pipelines, SageMaker, Azure ML, Databricks ML, NLP, RAG pipelines, Agentic LLM systems, chatbot development, DBT, data quality frameworks

PROFESSIONAL EXPERIENCE

GE Healthcare (Data Engineer) June 2024 - Present

● Designed Databricks Lakehouse architecture on AWS S3 with Unity Catalog as governance layer, enforcing centralized access control across catalogs, schemas, and tables, reducing unauthorized access incidents by 90 percent through policy enforcement at query time.

● Implemented Auto Loader ingestion using AWS S3 cloud Files source with file notification mode and checkpoint-based state tracking, enabling exactly once ingestion of 250 million records per day with resilient recovery from failures.

● Built Medallion architecture using Delta Lake transaction log-based storage across bronze, silver, and gold layers, ensuring ACID compliance, schema evolution control, and improving data consistency by 45 percent.

● Optimized Spark execution by analyzing Catalyst optimizer stages and DAG execution flow in Databricks, tuning shuffle boundaries, skew handling, and broadcast joins, reducing job runtime from 42 minutes to 16 minutes and improving compute efficiency by 62 percent.

● Implemented incremental processing using Delta MERGE with condition based upserts and schema aware transformations, eliminating full reload pipelines and reducing compute cost by 55 percent.

● Designed event driven orchestration using AWS Step Functions integrated with AWS Lambda for validation, branching, and retry logic, reducing pipeline recovery time from 45 minutes to 12 minutes.

● Built Structured Streaming pipelines using Spark streaming engine with watermarking, state store checkpointing, and event time windows, enabling processing of late arriving data with latency under 90 seconds.

● Integrated AWS Glue Data Catalog with Databricks Unity Catalog to synchronize metadata definitions and enable unified schema discovery across batch and streaming pipelines.

● Implemented schema governance using AWS Glue Schema Registry with pre ingestion validation and contract enforcement, reducing schema drift failures by 80 percent.

● Designed S3 data lake storage with optimized partitioning strategy, file compaction, and lifecycle policies, reducing S3 scan cost and improving Athena query efficiency by 28 percent.

● Enabled ad hoc analytics using AWS Athena over curated S3 datasets, eliminating need for compute clusters for validation and exploratory workloads.

● Built data quality framework using Great Expectations integrated with Databricks jobs and AWS CloudWatch logging for validation monitoring, reducing downstream data defects by 40 percent.

● Implemented observability layer using AWS CloudWatch metrics, Databricks job logs, and Lambda alerting for pipeline health tracking and anomaly detection.

● Implemented end to end data lineage using Open Lineage integrated with Spark jobs and orchestration workflows, enabling full traceability across ingestion, transformation, and serving layers. Farmers Insurance (Data Engineer) Jun 2022 - Aug 2023

● Migrated 120 SSIS ETL workflows into AWS Glue and Step Functions by decomposing monolithic control flow packages into modular Spark based tasks, reducing ETL runtime by 52 percent.

● Tuned SQL Server stored procedures across 40 million records using execution plan analysis, indexing optimization, and elimination of key lookups, reducing runtime from 18 minutes to 6 minutes.

● Implemented AWS DMS CDC replication using SQL Server transaction log capture, streaming inserts, updates, and deletes into S3 with sub 5 second latency and high consistency.

● Designed S3 data lake architecture storing 3 to 5 terabytes daily using partitioned parquet format, enabling efficient querying via Athena and Redshift Spectrum.

● Built AWS Glue ETL pipelines using Spark execution engine with dynamic frame transformations, predicate pushdown, and optimized joins, improving processing efficiency and reducing scanned data volume by 55 percent.

● Designed Amazon Redshift data warehouse using star schema modeling with optimized distribution keys and sort keys aligned to query patterns, reducing cross node data movement by 60 percent and improving BI query performance by 58 percent.

● Integrated AWS Athena for serverless querying over S3 data lake to reduce Redshift compute dependency and improve cost efficiency for ad hoc analytics.

● Replaced SSIS orchestration with AWS Step Functions state machines implementing retry, parallel execution, and failure handling logic, improving reliability and reducing recovery time by 70 percent.

● Built AWS Lambda based validation framework for pre and post ETL checks, ensuring data quality enforcement and automated failure alerting.

● Implemented SCD Type 2 logic using AWS Glue Spark jobs and Redshift staging tables with surrogate keys and effective dating for full historical tracking.

● Built reconciliation framework using Python and AWS Glue comparing source SQL Server and Redshift datasets using hash-based validation, reducing mismatch rate from 6 percent to under 1 percent.

● Implemented pipeline monitoring using AWS CloudWatch logs and metrics, enabling faster failure detection and operational visibility.

● Improved data freshness SLA from 12 hours to 4 hours by converting batch ETL pipelines into CDC driven incremental ingestion workflows.

Yes Bank (Data Engineer) Jan 2021 - Jun 2022

● Built Azure Data Factory ingestion framework integrating multiple core banking systems into Azure Data Lake Storage and Azure SQL using metadata driven pipeline design with externalized dataset definitions and transformation rules, processing 120 million transactions daily with 99.8 percent success rate.

● Designed Azure Data Lake Storage Gen2 with hierarchical raw, curated, processed zones using parquet storage with partition pruning, compression, and ingestion-driven partitioning, processing 2 to 3 terabytes daily while improving query performance by 60 percent and reducing Synapse scan cost by 40 percent.

● Implemented real time ingestion using Azure Event Hubs with partition key-based distribution and consumer group isolation, integrated with Azure Functions for event driven processing, achieving 5 to 8 second end to end latency.

● Built Azure Synapse dedicated SQL pool using hash distributed facts and replicated dimensions to reduce data movement, optimized via distribution tuning, materialized views, skew reduction, and caching, reducing query time from 22 minutes to 9 minutes and improving performance by 52 percent.

● Implemented Spark based ELT pipelines in Synapse Spark pools using DAG execution, lazy evaluation, and columnar pushdown optimization, reducing batch processing window from 10 hours to 3.5 hours.

● Integrated Azure Blob Storage with ADLS Gen2 to support tiered storage architecture for hot and cold datasets, improving storage lifecycle efficiency and cost optimization.

● Built metadata driven Azure Data Factory framework across 40 plus pipelines using configuration-based orchestration, reducing development effort by 50 percent and improving deployment consistency.

● Implemented SCD Type 2 framework using surrogate keys and system versioned tables in Azure SQL and Synapse, enabling full historical tracking of customer and account changes.

● Integrated Azure Key Vault with Azure Data Factory and Synapse using managed identity authentication, eliminating hardcoded credentials and improving security compliance by 90 percent.

● Built centralized monitoring framework using Azure Monitor, Log Analytics, and KQL dashboards with retry policies, dependency orchestration, and checkpoint recovery, reducing pipeline failure rate from 12 percent to 3 percent and improving observability.

● Improved data reconciliation accuracy from 93 percent to 99.6 percent using checksum-based validation, schema checks, and record level verification across systems.

● Improved regulatory reporting efficiency by building optimized Synapse data marts with pre aggregated views and lineage aware transformations, reducing report generation time from 6 hours to 1.5 hours. EDUCATION

Master’s in Business Analytics (University of Houston)

Contact this candidate