Data Engineer

Shashank G — Data Engineer

303-***-**** *****************@*****.***

PROFESSIONAL SUMMARY:

4+years of experience as a Data Engineer in designing, developing, implementing, and maintaining scalable batch and real- time data processing solutions across Healthcare, Banking, and Retail domains.

Hands-on experience with Big Data technologies including Apache Spark, PySpark, Hadoop, Hive, Snowflake, Databricks, and HDFS for processing large-scale structured and semi-structured datasets.

Strong experience with AWS Cloud services including Amazon S3, EMR, Glue, Lambda, IAM, Redshift, Athena, CloudWatch, EC2, and RDS, and Microsoft Azure services including Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure SQL Database, Azure Key Vault, and Azure Monitor.

Proficient in programming using Python, PySpark, Scala, and SQL for developing ETL pipelines, distributed data processing applications, and data engineering solutions.

Extensive experience writing advanced SQL, optimizing complex queries, developing stored procedures, and working with Snowflake SQL, HiveQL, Oracle SQL, PostgreSQL, and SQL Server.

Experience implementing CI/CD pipelines using Jenkins, containerizing applications with Docker, and deploying workloads on Kubernetes for scalable data platforms.

Knowledge of Machine Learning workflows using AWS SageMaker and Python libraries to prepare analytical datasets for predictive modeling.

Hands-on experience scheduling and orchestrating workflows using Apache Airflow, AWS Step Functions, and Azure Data Factory Pipelines.

Experience working with on-premise Hadoop ecosystems including Apache Hadoop, Hive, HDFS, YARN, and Sqoop for enterprise-scale data processing.

Strong experience designing ETL/ELT pipelines using AWS Glue, Azure Data Factory, Informatica PowerCenter, and implementing dimensional data models for enterprise reporting.

Experience processing multiple data formats including CSV, JSON, XML, Avro, ORC, Parquet, and Text files across distributed platforms.

Hands-on experience developing analytical datasets and dashboards using Power BI and Tableau to support business intelligence and executive reporting.

Experienced in preparing technical design documents, architecture diagrams, data lineage documentation, and operational runbooks using Confluence, SharePoint, and Lucidchart.

Familiar with OLAP technologies and multidimensional reporting concepts using SQL Server Analysis Services (SSAS).

Proficient with version control systems including Git and GitHub for source code management and collaborative development.

Strong understanding of Software Development Life Cycle (SDLC) methodologies including Agile Scrum, requirement analysis, solution design, development, testing, deployment, and production support.

Experience working with Agile project management tools including JIRA for sprint planning, backlog management, defect tracking, and release coordination.

Excellent analytical, communication, and problem-solving skills with the ability to collaborate effectively across cross- functional teams and deliver high-quality data engineering solutions within project timelines. WORK EXPERIENCE:

Data Engineer @ Aetna Hartford, CT May 2024 – Present

Designed and implemented a scalable AWS-based data lake architecture to centralize healthcare claims, provider, member, and pharmacy data for enterprise analytics.

Collaborated with business analysts, solution architects, and cross-functional teams to gather business requirements and design end-to-end ETL solutions for healthcare reporting.

Developed scalable PySpark applications on Amazon EMR to process large volumes of structured and semi-structured healthcare datasets.

Built reusable Spark transformations for data cleansing, standardization, enrichment, aggregation, and validation to support downstream analytical applications.

Developed automated ETL pipelines using AWS Glue to ingest data from Oracle databases, REST APIs, CSV, JSON, and Avro files into Amazon S3.

Implemented incremental data loading strategies, partitioning, and optimized Parquet file formats to improve storage and query performance.

Designed and optimized Snowflake schemas, ELT pipelines, materialized views, and clustering strategies to support enterprise reporting and analytics.

Developed near real-time data ingestion pipelines using Apache Kafka integrated with Spark Streaming for processing healthcare events and transactional data.

Utilized AWS Lambda to automate metadata validation, event-driven file processing, workflow triggering, and notification services.

Configured and maintained Apache Airflow DAGs to orchestrate batch workflows, manage task dependencies, schedule pipelines, and monitor job execution.

Implemented data quality validation, reconciliation processes, and exception handling to ensure accurate and reliable data before loading into analytical repositories.

Configured centralized logging and monitoring using Amazon CloudWatch and Log4j to troubleshoot pipeline failures and support production operations.

Secured sensitive healthcare data using AWS IAM, encryption at rest and in transit, role-based access control, and HIPAA- compliant security practices.

Containerized PySpark applications using Docker, deployed workloads on Kubernetes, and automated CI/CD pipelines using Jenkins while managing source code through GitHub.

Published curated datasets through Amazon Athena and Power BI, participated in Agile Scrum ceremonies using JIRA, performed code reviews, supported production deployments, and optimized ETL workflows for business reporting. Technologies Used: Python, PySpark, Apache Spark, AWS EMR, Amazon S3, AWS Glue, AWS Lambda, Athena, Amazon Redshift, Snowflake, Apache Kafka, Apache Airflow, Docker, Kubernetes, DynamoDB, Jenkins, GitHub, Amazon CloudWatch, IAM, Parquet, JSON, Avro, Power BI, JIRA.

Data Engineer @ Morgan Stanley New York, NY Nov 2021 – Dec 2022

Designed and implemented scalable data engineering solutions on the Azure platform to support enterprise risk management, regulatory reporting, and financial analytics.

Collaborated with business analysts and solution architects to gather business requirements and develop end-to-end data integration workflows.

Developed high-performance ETL pipelines using Spark Scala to process large volumes of financial and transactional data.

Built distributed Spark applications for cleansing, aggregating, and transforming datasets using partitioning, caching, and optimization techniques.

Developed automated data ingestion pipelines using Azure Data Factory (ADF) to extract data from Oracle, SQL Server, REST APIs, and flat files into Azure Data Lake Storage Gen2 (ADLS Gen2).

Migrated legacy on-premise ETL workloads and enterprise datasets to Azure to support scalable cloud-based analytics.

Designed and implemented enterprise data warehouse solutions using Azure Synapse Analytics by developing fact tables, dimension tables, and optimized data models.

Integrated Apache Kafka with Spark Streaming to process real-time market events and transaction feeds for downstream analytical applications.

Implemented data validation, reconciliation, and quality checks to ensure consistency and accuracy before loading data into analytical repositories.

Configured centralized logging, exception handling, and operational monitoring using Azure Monitor and Log4j for production support.

Secured enterprise data using Azure Key Vault, role-based access control (RBAC), managed identities, and encryption policies to comply with organizational security standards.

Developed and scheduled batch workflows using Apache Airflow, managing task dependencies and automating end-to-end data pipelines.

Containerized Spark applications using Docker, deployed workloads on Kubernetes, and automated CI/CD processes using Jenkins.

Managed source code using GitHub, participated in Agile Scrum ceremonies using JIRA, performed peer code reviews, and supported production deployments.

Published curated datasets through Azure Synapse Analytics for business users and developed interactive dashboards using Tableau for financial reporting and analytics.

Technologies Used: Scala, Spark Scala, Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Synapse Analytics, Azure SQL Database, Azure Key Vault, Azure Monitor, Apache Kafka, Apache Airflow, Docker, Kubernetes, Cassandra, Jenkins, GitHub, Tableau, Parquet, JSON, CSV, JIRA. Junior Data Engineer @ Meijer Grand Rapids, MI Feb 2021 – Oct 2021

Participated in the design and development of enterprise data integration solutions to support retail sales, inventory management, merchandising, and customer analytics.

Collaborated with business analysts to understand reporting requirements and translate business needs into scalable ETL and data warehouse solutions.

Developed ETL workflows using Informatica PowerCenter to extract, transform, and load data from Oracle and MySQL into the Hadoop ecosystem.

Built data ingestion processes using Sqoop to transfer transactional data from relational databases into HDFS for analytical processing.

Developed optimized Hive tables and HiveQL queries for data transformation, aggregation, and reporting.

Worked with Hadoop HDFS to store and process structured and semi-structured retail data, including sales, inventory, customer, and supplier information.

Processed data from multiple file formats including CSV, XML, JSON, and Parquet, implementing validation and transformation rules during data ingestion.

Created reusable Unix Shell scripts to automate ETL executions, perform file validation, archive processed data, and generate job status notifications.

Implemented data quality checks, reconciliation processes, and exception handling to ensure accurate and consistent data across downstream systems.

Configured workflow scheduling using Oozie to automate daily and weekly batch processing jobs and manage workflow dependencies.

Developed advanced SQL queries, stored procedures, and performance tuning techniques on Oracle and MySQL databases.

Utilized HBase to store and retrieve high-volume operational data for low-latency access in retail reporting applications.

Implemented application logging and error tracking using Log4j to support troubleshooting and production issue resolution.

Managed source code using Git, supported deployment activities through Jenkins, and actively participated in Agile Scrum ceremonies using JIRA.

Delivered curated datasets to business users through Tableau dashboards for sales analysis, inventory reporting, and executive decision-making.

Technologies Used: Python, SQL, Informatica PowerCenter, Hadoop, HDFS, Hive, Sqoop, Oozie, Oracle, MySQL, HBase, Unix Shell Scripting, Jenkins, Git, Tableau, Log4j, CSV, XML, JSON, Parquet, JIRA. TECHNICAL SKILLS:

Languages: Python, PySpark, Scala, SQL, HiveQL, PL/SQL, Shell Scripting

Big Data Technologies: Apache Spark, Hadoop, HDFS, Hive, YARN, Databricks, Snowflake, Sqoop, Apache Flume

Cloud Technologies: AWS (S3, EMR, EC2, Glue, Lambda, Athena, Redshift, RDS, IAM, CloudWatch, SNS, SQS, Kinesis, Secrets Manager), Azure (Azure Data Factory, ADLS Gen2, Azure Synapse Analytics, Azure SQL Database, Azure Key Vault, Azure Monitor, Azure Functions, Event Hubs)

Databases: Snowflake, Oracle, PostgreSQL, SQL Server, Amazon Redshift, Hive, MySQL

NoSQL Databases: DynamoDB, MongoDB, HBase

Streaming Technologies: Apache Kafka, Amazon Kinesis

ETL & Data Integration: AWS Glue, Azure Data Factory (ADF), Informatica PowerCenter

Data Warehousing: Snowflake, Amazon Redshift, Azure Synapse Analytics

Scheduling & Orchestration: Apache Airflow, AWS Step Functions, Azure Data Factory Pipelines, Cron

Data Formats: CSV, JSON, XML, Avro, ORC, Parquet, TXT

DevOps & CI/CD: Jenkins, Docker, Kubernetes

Version Control: Git, GitHub

Operating Frameworks: Apache Spark SQL, Spark Core, Spark DataFrames, Spark Streaming

Data Quality & Testing: Great Expectations, PyTest, Unit Testing, Data Validation, Reconciliation Testing

Logging & Monitoring: Log4j, Amazon CloudWatch, Azure Monitor

Security: AWS IAM, Azure Key Vault, Encryption at Rest, Encryption in Transit, Row-Level Security, Column-Level Security

BI & Visualization: Power BI, Tableau

Machine Learning: AWS SageMaker, Scikit-learn, Pandas, NumPy

Documentation Tools: Confluence, SharePoint, Lucidchart

Project Management & Ticketing: JIRA, ServiceNow

Methodologies: Agile Scrum, SDLC, Waterfall, CI/CD

Data Modeling: Star Schema, Snowflake Schema, Dimensional Modeling, ER Modeling EDUCATION:

Master of Science in Information Systems @ University of Colorado Denver

Contact this candidate