Azure Data Engineer

Harshavardhan Y

Phone: 937-***-****

Email: ******************@*****.***

Summary:

Accomplished Azure Data Engineer with over 9 years of diverse experience in the IT industry, adept at steering data engineering and big data projects. Proficient in orchestrating Azure Big Data and data transformation initiatives in banking, financial services, and healthcare sectors. Expert in crafting and deploying solutions using Azure Data Lake, Azure Data Factory, Azure Databricks, and other Azure services.

Technical Skills:

Azure Technologies: Azure Data Lake, Azure Data Factory, Azure Databricks, Azure Synapse, Azure SQL, Azure DW, Azure Storage

Big Data Technologies: Hadoop, Spark, Kafka, HBase, Hive, Pig, Sqoop, Flume, Storm Programming Languages: Java, Python, Scala, HiveQL, SQL, PL/SQL, JavaScript CI/CD Tools: Docker, Kubernetes, Jenkins

Databases: Oracle, SQL Server, MySQL, Teradata, MongoDB, Cassandra Data Visualization: Tableau, Power BI

Other Tools: Erwin, Informatica, SSIS, SSRS, Data modeling, ETL Professional Experience:

Sr. Data Engineer - Bank of Montreal

05/2021 - Present

- Directed the implementation of Azure Data Factory, Azure Databricks, Azure SQL, Azure DW, and Azure Synapse projects.

- Engineered and optimized ETL pipelines to enhance data transformation and integration processes.

- Streamlined CI/CD pipelines using Jenkins, boosting the efficiency of data engineering solutions.

- Leveraged Scala, Python, and Hadoop frameworks to process and analyze extensive datasets. Sr. Data Engineer - TWC

07/ 2019 - 04/ 2021

- Spearheaded end-to-end data engineering projects using Azure Databricks, Azure Data Factory, Azure SQL, and Azure DW.

- Utilized Cloudera, Apache Hadoop, HDFS, Hive, MapReduce, Cassandra, Oozie, Flume, Spark, SQL, and Scala.

- Developed and refined data pipelines for both real-time and batch data processing.

- Collaborated with cross-functional teams to integrate data solutions with business applications. Sr. Data Engineer/Analyst - New York HealthCare

04/2017 – 10/2018

- Capitalized on Azure Databricks, Azure Data Lake, Azure Stream, Azure ADF, and Azure DW for advanced data analytics and processing.

- Focused on ETL processes and established CI/CD pipelines for automated deployments.

- Employed big data technologies such as Spark, YARN, Hive, and Hadoop to manage and analyze large-scale datasets.

- Delivered data insights and analytics to support strategic decision-making processes. Sr. Data Analyst - Change Healthcare

08/2015 – 03/2017

- Exploited Teradata, Azure Databricks, Spark, SSIS, SQL, and Informatica Power Center for data integration and analysis.

- Concentrated on data modeling, ETL processes, and database management.

- Led data migration projects from on-premises to Azure cloud environments.

- Designed and maintained data models to support business intelligence and analytics needs. Data Analyst

Charles Schwab – Westlake, TX

05/2014 to 07/2015

- Performed data analysis and profiling of source data.

- Created and converted data models using Erwin.

- Designed source-to-target mappings for Teradata using Informatica Power Center.

- Conducted data cleansing and integration using Informatica Data Quality (IDQ).

- Developed Star and Snowflake Schemas for data warehousing.

- Assisted ETL team with documenting transformation rules.

- Conducted data analysis using SQL on Teradata and Oracle systems.

- Optimized and tuned SQL queries.

- Provided Physical Data Model training for ETL Developers. Education:

- Master’s Degree in Computer Science

- Wright State University, June 2019

- CGPA: 3.5

- Bachelor’s Degree in Computer Science

- VIT University

- GPA: 7.5

Certifications:

- Certified Spark & Hadoop Developer (CCA-175)

- Tableau Desktop Certified

- Azure Fundamentals (AZ-900)

- Azure Data Fundamentals (DP-900)

- AWS Cloud Practitioner

Key Projects:

- Lambda Architecture Implementation: Engineered and executed large-scale Lambda architectures using the Azure Data platform.

- Real-Time Data Processing: Developed real-time data processing and analytics solutions with Spark-SQL in Databricks.

- Data Visualization: Crafted complex data visualizations and dashboards using Tableau for comprehensive data sets.

- Database Migration: Directed the migration of SQL databases to Azure Data Lake, Azure SQL Database, Data Bricks, and Azure SQL Data Warehouse. Additional Experience:

- Data Modeling & Analysis: Proficient in Dimensional Data Modeling, Relational Data Modeling, Star Schema, Snowflake Modeling, FACT & Dimensions tables.

- Big Data Infrastructure: Extensive knowledge of Big Data infrastructure, distributed file systems

(HDFS), and parallel processing (MapReduce framework).

- NoSQL Data Models: Designed NoSQL data models for MongoDB and Azure Cosmos. Key Achievements:

- Optimized data pipeline performance, resulting in a 30% increase in data processing speed.

- Successfully led a team of data engineers to deliver critical data migration projects ahead of schedule.

- Improved data accuracy and consistency by implementing robust data validation protocols.

Contact this candidate