Data Engineer Azure

S MADHUKAR REDDY

Data Engineer

945-***-**** *********.**@*****.*** LinkedIn

PROFESSIONAL SUMMERRY

Certified Data Engineer with 4+ years of experience in optimizing ETL pipelines and building scalable data solutions on AWS and Azure, achieving up to 25% faster data processing and a 20% boost in efficiency. Skilled in Python, Apache Spark, SQL, and Snowflake, with a track record of enhancing data accessibility and troubleshooting complex data flows in Agile.

Professional EXPERIENCE

Data Engineer Axalta Apr 2023 to Present PA, US

Built Kafka-based real-time data pipelines and utilized Spark on Databricks for big data handling, reducing system latency by 30% and improving scalability for high-volume streaming data.

Developed complex ETL workflows in Azure Databricks, increasing data quality by 35% and boosting sales forecasting accuracy by 20%, leveraging SCD Type 1 and Type 2 in Delta Lake for historical tracking.

Designed and streamlined Azure Data Factory and Databricks pipelines, processing over 10 million records daily, achieving a 30% efficiency boost and reducing ingestion time by 40%.

Supported Power BI dashboards for sales trends and KPIs, driving faster decision-making across 10+ departments, implemented real-time monitoring in Azure Data Factory, reducing pipeline failures by 20% and enhancing system stability.

Data Engineer Cybermatic Solutions Oct 2020 to Aug 2021 India

Migrated over 5,00,00,000 records from an on-prem SQL Server to Azure SQL Database via Azure Data Factory, achieving a 30% improvement in query response time and enhanced system scalability.

Designed and implemented ADF ETL pipelines with Change Data Capture, increasing data load efficiency by 30% and streamlining data extraction, transformation, and loading into Azure Data Lake Gen2.

Automated complex SQL transformations with dbt’s Jinja templating, reducing manual ETL intervention and achieving 98% data accuracy through rigorous testing and validation, minimizing post-migration discrepancies.

Provisioned Azure SQL Database and configured secure, compliant connections in ADF, while documenting and training, leading to a 15% decrease in post-migration support requests.

Data Engineer Maxpi Tech Pvt Ltd Oct 2016 to Aug 2018 India

Designed a scalable ETL pipeline using AWS Glue, Apache Airflow, and Snowflake, reducing data processing time by 30% and enabling real-time analytics for stakeholders.

Developed and optimized AWS Glue jobs and 15+ crawlers, improving ETL efficiency by 30% and lowering infrastructure costs by 15%. Integrated Airflow for automated workflows, boosting data load frequency by 40%.

Managed metadata for over 50TB of data using AWS Glue Data Catalog, achieving 60% faster querying within Snowflake, while maintaining a 90% success rate for Airflow job executions with error handling and retries, reducing pipeline downtime by 30%.

Leveraged Amazon Kinesis, Lambda, and Redshift for a serverless real-time data ingestion system, cutting latency by 30% and costs by 20%.

Technical Skills

Programming Languages: Python, SQL, Shell Scripting, Java.

Cloud Platforms: Azure, AWS, GCP.

Databases: MS SQL Server, MySQL, PostgreSQL, NoSQL MongoDB, Cassandra, Redshift, S3 and Data Lake.

Big Data Technologies & ETL: Snowflake, Databricks, DBT, Pyspark, Airflow, ADF, AWS Glue.

Python Libraries: Pandas, NumPy, SciPy, Matplotlib.

CI/CD and Version Control Tools: Git, GitHub, Kubernetes(K8S), DevOps.

SDLC Methodologies: Agile/SCRUM, Waterfall

Certifications

Microsoft Certified: Azure Data Engineer Associate (DP-203).

Education

Masters in science and technology Lamar University Beaumont, Texas, USA (2022)

Contact this candidate