Jaya Gopal Chodagiri
+1-972-***-**** ************@*****.***
Dallas, Texas – 75056
Professional summary:
Senior Data Engineer with 7+ years of experience designing, building, and optimizing cloud-native data platforms and scalable ETL/ELT pipelines across Azure and AWS ecosystems.
Expertise in big data processing frameworks including Apache Spark, PySpark, Databricks, and Airflow for large-scale data transformation and orchestration.
Strong experience in developing high-performance data pipelines using Azure Data Factory (ADF), AWS Glue, and Snowflake, handling millions of records daily.
Strong expertise in Microsoft Azure ecosystem and Microsoft Fabric, including Fabric Lakehouse, Data Factory (Fabric), Dataflows Gen2, and OneLake for unified data architecture.
Hands-on experience in designing modern lakehouse architectures using Azure Databricks and Microsoft Fabric, enabling scalable analytics and real-time data processing.
Experience leveraging Microsoft Fabric for end-to-end data engineering, data integration, and Power BI-driven analytics within a unified SaaS platform.
Proficient in data modeling, data warehousing, and lakehouse architectures using ADLS Gen2, Azure Synapse, Snowflake, and Redshift.
Hands-on expertise in CI/CD pipelines, DevOps practices, and Infrastructure as Code (IaC) using Azure DevOps, Jenkins, Git, and Docker.
Skilled in implementing data security, governance, and compliance frameworks, including RBAC, data masking, encryption, and HIPAA/PII standards.
Experience in building real-time and batch data processing solutions integrating Kafka streams, APIs, and cloud-native services.
Strong background in performance tuning, query optimization, and distributed computing to enhance pipeline efficiency and reduce processing time.
Expertise in enabling AI/ML workflows by preparing feature-rich datasets and supporting model training pipelines.
Proven ability to collaborate with data scientists, analysts, and business stakeholders to deliver scalable and business-driven data solutions.
Experience in developing interactive dashboards and data visualization solutions using Power BI and Tableau.
Adept at working in Agile/Scrum environments, managing end-to-end data engineering lifecycle from ingestion to analytics.
Strong experience in cloud migration, modernization, and legacy system transformation initiatives.
Skilled in building automated workflows and orchestration frameworks to improve operational efficiency and reduce manual intervention.
Self-driven and results-oriented professional with a focus on delivering high-quality, scalable, and secure data solutions in enterprise environments.
Technical Skills:
Programming & Query Languages: Python, SQL, Scala, PL/SQL, Shell Scripting
Cloud & Data Platforms: Microsoft Azure (ADF, ADLS Gen2, Synapse, Azure SQL, Logic Apps, Functions), Microsoft Fabric (Lakehouse, Data Factory, Dataflows Gen2, OneLake, Power BI), AWS (S3, EC2, Glue, Lambda, RDS), Snowflake
Data Engineering & Big Data: Apache Spark, PySpark, Databricks, Apache Airflow, ETL/ELT Pipelines, Data Lakes, Data Warehousing
Databases & Storage: Azure SQL Database, AWS RDS, MongoDB, Cassandra, DynamoDB
Visualization & Analytics: Power BI, Tableau
DevOps & CI/CD: Azure DevOps, Jenkins, Git, GitHub, Docker, Infrastructure as Code
Data Security & Governance: RBAC, Data Masking, Encryption, HIPAA/PHI/PII Compliance, Azure Key Vault, Azure AD
Methodologies & Tools: Agile/Scrum, Eclipse, PyCharm, Maven
Work Experience:
Client: UnitedHealth Group Dallas, TX Jun 2024 - Present
Role: Senior Azure & Microsoft Fabric Data Engineer
Leveraged Azure Cloud, Snowflake, and Databricks to design and automate end-to-end ETL pipelines in ADF, processing ~4M medical claims daily with enhanced reliability.
Optimized ETL processes using PySpark jobs in Azure Databricks for data cleansing, validation, and large-scale transformations, reducing runtime by 60%.
Designed and implemented Microsoft Fabric Lakehouse architecture to unify data ingestion, transformation, and analytics using OneLake and Dataflows Gen2.
Leveraged Fabric Data Factory and pipelines to orchestrate end-to-end data workflows, improving data integration efficiency and reducing latency.
Integrated Microsoft Fabric with Power BI to deliver real-time, business-ready datasets and enterprise reporting solutions.
Migrated data warehouse from Snowflake to Databricks, rewriting SQL logic into Spark-based transformations to boost scalability and cost efficiency.
Implemented robust data security measures including row-level security (RLS), data masking, and RBAC using Azure Key Vault and AAD to safeguard PHI/PII data.
Developed a job orchestration framework with Logic Apps and Azure Functions to streamline workflows and improve operational efficiency.
Collaborated with analytics teams on AI/ML enablement by preparing ML-ready datasets and feature pipelines in PySpark, supporting predictive modeling.
Integrated curated data and ML outputs into Power BI dashboards to provide real-time and predictive intelligence for business stakeholders.
Managed deployments via Azure DevOps and GitHub using infrastructure as code templates to ensure streamlined CI/CD processes and maintain version control in an Agile environment.
Environment: Microsoft Fabric (Lakehouse, Data Factory, Dataflows Gen2, OneLake, Power BI), Azure Data Factory (ADF), Azure Databricks, PySpark, Snowflake, ADLS Gen2, Azure SQL, Azure Logic Apps, Azure Functions, Azure DevOps, GitHub, Docker, Azure Key Vault, Azure Active Directory (AAD), CI/CD, Agile/Scrum
Client: Verizon Dallas, TX Feb 2023 - May 2024
Role: Senior Azure Data Engineer
Engineered and deployed data pipelines using Azure Data Factory and Databricks for customer data processing, enhancing data integration speed and quality.
Collaborated with cross-functional teams, including engineers and data scientists, to build a Customer Data Platform (CDP) managing over 1.5M daily interactions, resulting in an 18% increase in user engagement.
Developed Spark jobs on Databricks to cleanse, standardize, and transform high-volume datasets from SQL Server and Kafka streams, securing data accuracy.
Supported modern data platform evolution towards Azure-native and Fabric-ready architecture for scalable analytics.
Delivered real-time analytics via Power BI, reducing query times by 40% and enabling immediate, data-driven decision-making.
Automated over 50 workflows using Python and Logic Apps, significantly reducing manual effort by 25% and improving pipeline reliability.
Implemented CI/CD pipelines in Azure DevOps and containerization with Docker to ensure smooth deployments and stable production releases.
Supported AI/ML initiatives by building and optimizing training datasets, thereby improving model accuracy and reducing retraining cycles.
Integrated real-time data streams into recommendation pipelines to power AI-driven personalization and targeted marketing strategies.
Enhanced business intelligence by integrating ML model outcomes into Power BI dashboards, shifting analytics from descriptive to predictive.
Environment: Azure Data Factory (ADF), Azure Databricks, PySpark, Apache Spark, Kafka, Azure SQL, Power BI, Python, Azure DevOps, Docker, Logic Apps, CI/CD Pipelines, Git, Agile/Scrum
Client: Cleveland State University Cleveland, OH Aug 2021 - Dec 2022
Role: Graduate Assistant
Supervised laboratory sessions, workshops, and academic activities, providing consistent guidance to students.
Designed and built interactive Power BI dashboards to visualize key academic and operational metrics, driving data-informed decisions.
Provided technical support to faculty and students by resolving issues from support tickets, ensuring smooth operation of academic systems.
Assisted in the transition and development of the college website during the CSU merger, ensuring seamless integration of resources and information.
Environment: Power BI, SQL, Excel, Data Visualization Tools, Academic Systems, Windows Environment
Client: Veritas Soft Solutions, Hyderabad, India. May 2017 - Jul 2021
Role: Data Engineer
Designed, implemented, and maintained production-grade data pipelines on AWS using services such as S3, EC2, Glue, and Lambda, ensuring efficient data integration and processing.
Leveraged Apache Airflow and Python to orchestrate and optimize data operations and processing workflows, enhancing scalability and efficiency.
Implemented data validation and cleansing routines with Python libraries including Pandas and PySpark, improving data quality and consistency across large datasets.
Developed SQL Server tables, views, stored procedures, and functions to support downstream analytics and reporting requirements.
Conducted performance tuning and implemented monitoring solutions for data pipelines, achieving improved query performance and resource utilization.
Integrated on-premises legacy databases with cloud services to support both real-time and batch processing, boosting system reliability by 20%.
Guided technical teams on data engineering best practices and methodologies to ensure aligned, secure, and efficient data solutions.
Led an initiative to automate AWS EC2 resource management, optimizing costs by 15% for critical projects.
Refactored Python code to optimize data pipeline performance, reducing processing time and resource demands in cloud environments.
Conducted data modeling and schema design for operational databases, resulting in a 30% improvement in query performance.
Developed solutions for loading data into a Snowflake data lake environment from AWS EMR, enabling business users and data scientists to leverage Tableau/OBIEE for advanced analytics.
Created AWS Glue jobs and managed catalog table creation with crawlers and manual configurations to facilitate efficient ETL processes.
Implemented a CI/CD pipeline using GitHub and AWS, supporting continuous enhancements and reliable production deployments.
Built and automated the creation of S3 data lake infrastructure with AWS Lambda functions and API Gateways, streamlining subsequent ETL processes to a Cloud warehouse (Redshift) for advanced analytics.
Developed Snow pipes for continuous data injection using AWS S3 event handlers, ensuring timely and seamless data flow.
Environment: AWS (S3, EC2, Glue, Lambda, RDS), Apache Airflow, PySpark, Python, SQL Server, Snowflake, Snowpipe, Redshift, API Gateway, MongoDB, Cassandra, DynamoDB, GitHub, CI/CD Pipelines, Linux, Agile