Senior Data Engineer (Data Engineering)

VIKAS REDDY AMARAVATHI

DATA ENGINEER SENIOR DATA ENGINEER

USA +1-940-***-**** *.*********@*****.*** LinkedIn: linkedin.com/in/vikas-reddy-a-avr03

SUMMARY

Data Engineer with 5+ years of experience designing, developing, optimizing, and operating scalable batch, streaming, ETL/ELT, data lake, lakehouse, and cloud data warehouse solutions across healthcare, financial services, and enterprise environments. Strong hands-on expertise in Python, SQL, PySpark, Apache Spark, Kafka, Airflow, Databricks, AWS, and Azure, with experience processing large-scale structured and semi-structured datasets. Proven experience developing reliable end-to-end data pipelines, optimizing distributed processing workloads, designing analytics-ready data models, and improving data quality, availability, latency, scalability, observability, and cloud cost efficiency. Experienced in event-driven architectures, workflow orchestration, data modeling, data validation, CI/CD, Infrastructure as Code, data governance, monitoring, and security controls. Skilled at transforming raw enterprise data into trusted, reusable datasets that support business intelligence, reporting, analytics, and downstream applications.

SKILLS

Programming & Query Languages: Python, SQL, PySpark, Scala, Bash/Shell, Java

Data Engineering & Big Data: Apache Spark, Spark SQL, Spark Structured Streaming, Apache Kafka, Hadoop, HDFS, Hive, Apache Flink, Apache NiFi, Sqoop

ETL/ELT & Workflow Orchestration: Apache Airflow, Azure Data Factory, AWS Glue, Databricks, dbt, Informatica, Talend

AWS: Amazon S3, AWS Glue, Amazon EMR, AWS Lambda, Amazon Redsachift, Amazon Kinesis, CloudWatch, CloudFormation

Azure: Azure Databricks, Azure Data Factory, Azure Data Lake Storage Gen2, Azure Synapse Analytics, Azure Event Hubs, Azure Monitor, Log Analytics

GCP: Google BigQuery, Dataproc, Pub/Sub, Cloud Storage

Databases & Data Warehousing: Amazon Redshift, Azure Synapse Analytics, Snowflake, BigQuery, PostgreSQL, MySQL, SQL Server, MongoDB, Cassandra

Data Lake & Lakehouse: Delta Lake, Apache Iceberg, Data Lake Architecture, Lakehouse Architecture

Data Modeling & Performance: Dimensional Data Modeling, Logical & Physical Data Modeling, Partitioning, Compression, Indexing, Caching, Distribution Strategies, Spark Optimization, SQL Query Tuning

Data Quality & Governance: Data Validation, Reconciliation, Data Quality Controls, Data Lineage, Great Expectations, RBAC, Encryption, Data Governance, HIPAA-Aware Data Processing

DevOps & Automation: Git, GitHub Actions, Terraform, Docker, Kubernetes, Jenkins, CI/CD, Infrastructure as Code

Development & Analytics: Databricks, Jupyter Notebook, Looker, Metabase

GCP: Google BigQuery, Google Cloud Storage (GCS), Google Cloud Dataproc, Pub/Sub, BigQuery SQL, Partitioning & Clustering

PROFESSIONAL EXPERIENCE

Optum — Senior Data Engineer Healthcare Texas, USA Jul 2025 – Present

Design and develop scalable Azure Data Factory and Azure Databricks ETL/ELT pipelines to ingest and process structured and semi-structured healthcare data from multiple enterprise systems into Azure Data Lake Storage Gen2 and Azure Synapse Analytics for reporting and analytics.

Support multi-cloud data engineering workloads across Azure and GCP, leveraging Google Cloud Storage (GCS), BigQuery, Dataproc, and Pub/Sub to ingest, process, transform, and analyze enterprise healthcare datasets.

Build reusable PySpark and SQL transformation frameworks for cleansing, standardization, validation, reconciliation, deduplication, and analytics-ready data preparation across large healthcare datasets.

Develop PySpark-based batch processing workloads on Google Cloud Dataproc, processing large datasets stored in GCS and publishing curated data into BigQuery for downstream reporting and analytical use cases.

Design and optimize BigQuery datasets and SQL transformations using partitioning, clustering, optimized joins, predicate filtering, and efficient query patterns to improve performance and control analytical compute costs.

Modernize legacy batch workflows by implementing near-real-time, event-driven ingestion using Azure Event Hubs, Apache Spark Structured Streaming, and PySpark, reducing data availability latency from approximately 4 hours to under 30 minutes.

Develop event-based ingestion patterns using Google Cloud Pub/Sub for selected GCP workloads and integrate incoming data with downstream processing and analytics pipelines.

Implement automated data quality, validation, and reconciliation controls across BigQuery, GCS, ADLS, and Synapse datasets, improving consistency and reliability of data consumed by reporting and analytics teams.

Optimize Azure Synapse Analytics workloads through partitioning, distribution strategy optimization, data modeling, and SQL query tuning, improving query performance by approximately 40% while reducing compute costs by 20%.

Optimize distributed Spark and Dataproc workloads through partition management, efficient joins, caching, memory tuning, and reduction of unnecessary data shuffling to improve processing performance and scalability.

Implement data governance and security controls, including RBAC, encryption, access management, and data quality standards, to support secure handling of sensitive healthcare information and HIPAA-aligned data processing requirements.

Develop production monitoring, centralized logging, and alerting using Azure Monitor and Log Analytics, achieving approximately 99.5% pipeline availability and reducing incident resolution time by more than 60%.

Capital One— Data Engineer Financial Services Texas, USA Jan 2024 – May 2025

Designed and developed scalable AWS data pipelines using Amazon S3, AWS Glue, AWS Lambda, PySpark, SQL, and Amazon Redshift to process high-volume transactional datasets supporting risk analytics, compliance reporting, and business intelligence.

Developed and maintained 20+ production ETL/ELT workflows to ingest, cleanse, normalize, transform, validate, and publish structured and semi-structured datasets into Redshift and S3-based data lake environments.

Contributed to multi-cloud data engineering initiatives across AWS and GCP, leveraging Google Cloud Storage, BigQuery, Dataproc, and Pub/Sub for selected ingestion, transformation, and analytical workloads.

Built scalable PySpark data processing pipelines on Google Cloud Dataproc to cleanse, enrich, aggregate, and transform high-volume datasets stored in GCS before publishing analytics-ready datasets into BigQuery.

Developed and optimized BigQuery SQL pipelines and data models, applying partitioning, clustering, optimized joins, incremental processing, and query tuning techniques to improve analytical performance and processing efficiency.

Implemented event-driven data ingestion using Amazon Kinesis, Apache Kafka, and AWS Lambda, reducing end-to-end data latency by approximately 60% and enabling near-real-time reporting.

Supported streaming ingestion patterns using Google Cloud Pub/Sub for selected event-driven workloads, integrating incoming events with downstream GCP processing and analytical datasets.

Developed cross-cloud data integration and validation workflows between AWS and GCP, supporting controlled movement and reconciliation of datasets across Amazon S3, Google Cloud Storage, Amazon Redshift, and BigQuery.

Designed and optimized analytics-ready Amazon Redshift datasets through compression, distribution strategies, data modeling, and advanced SQL tuning, contributing to approximately 20% reduction in cloud infrastructure costs.

Developed reusable PySpark and SQL transformation frameworks with standardized validation, exception handling, reconciliation, and data quality checks to improve reliability of downstream analytics datasets.

Automated infrastructure and data application deployments using Terraform, AWS CloudFormation, and GitHub Actions, supporting repeatable CI/CD and Infrastructure as Code (IaC) practices across cloud environments.

Implemented production pipeline monitoring, centralized logging, failure notifications, and operational troubleshooting using Amazon CloudWatch, improving visibility and reliability of business-critical data workflows.

Accenture — Data EngineerIT Services & Consulting Hyderabad, India Jan 2021 – May 2023

Built and maintained scalable batch and real-time data pipelines using Apache Spark, PySpark, Kafka, and Spark Streaming, processing more than 200M records per day and enabling near-real-time analytics and operational reporting.

Designed end-to-end ETL/ELT workflows using PySpark, Spark SQL, and SQL to ingest, cleanse, transform, integrate, and publish structured and semi-structured datasets from multiple enterprise source systems.

Developed reusable PySpark transformation frameworks for data cleansing, standardization, deduplication, enrichment, aggregation, and business-rule implementation, reducing data preparation time by approximately 35%.

Orchestrated and supported 30+ production data pipelines using Apache Airflow, implementing DAG development, workflow scheduling, dependency management, parameterization, retry mechanisms, monitoring, and failure handling.

Developed Kafka-based streaming pipelines to ingest and process high-volume event data, integrating Kafka topics with Spark-based processing workflows to support near-real-time downstream analytics.

Optimized Apache Spark applications using partitioning, broadcast joins, caching, memory tuning, data serialization, and distributed processing techniques, increasing processing throughput by approximately 45% while reducing infrastructure costs by 20%.

Worked with Hadoop, HDFS, and Hive-based data environments to store, process, and query large-scale enterprise datasets, developing optimized Hive and Spark SQL transformations for downstream analytical consumption.

Implemented automated data quality and validation controls, including record-count validation, null checks, duplicate detection, schema validation, source-to-target reconciliation, and exception handling to improve reliability of production datasets.

Developed optimized SQL and Spark SQL queries for complex joins, aggregations, transformations, and analytical processing while improving query performance across large-volume datasets.

Automated recurring data processing and operational workflows, reducing manual intervention by approximately 70% and improving consistency and reliability of production pipeline execution.

Implemented pipeline monitoring, logging, alerting, and troubleshooting procedures for Spark and Airflow workflows, performing root-cause analysis and resolving production data issues to improve overall pipeline availability.

Collaborated with data analysts, business stakeholders, application teams, and downstream consumers to translate data requirements into scalable pipelines and deliver reliable, reusable, analytics-ready datasets for reporting and business intelligence.

EDUCATION

Master of Science in Computer Science May 2025

University of North Texas — Denton, TX, USA

Bachelor’s Degree in Computer Science May 2023

Malla Reddy College — Hyderabad, India

Contact this candidate