Azure Data Engineer

Name: Deepika

Azure Data Engineer

Contact: +1-317-***-****

Email: *************@*****.***

LinkedIn: linkedin.com/in/deepikareddie3

Professional Summary

Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and Migrating On premise databases to Azure Data Lake store using Azure Data factory.

Experience in implementing Azure data solutions, provisioning storage accounts, Azure Data Factory, SQL server, SQL Databases, SQL Data warehouse, Azure Data Bricks and Azure Cosmos DB.

Experience in Data extraction (extract, Schemas, corrupt record handling and parallelized code), transformations and loads

(user - defined functions, join optimizations) and Production (optimize and automate Extract, Transform and Load).

Have good experience in designing cloud-based solutions in Azure by creating Azure SQL database, setting up Elastic pool jobs and designing tabular models in Azure analysis services.

Involved with the Design and Development of ETL process related to benefits and offers data into the data warehouse from different sources.

Possess strong Documentation skill and knowledge sharing among Team, conducted data modeling sessions for different user groups, facilitated common data models between different applications, participated in requirement sessions to identify logical entities.

Excellent knowledge in preparing required project documentation and tracking and reporting regularly on the status of projects to all project stakeholders.

Experience in UNIX shell scripting for processing large volumes of data from varied sources and loading into databases like Teradata.

Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka, and Flume.

Proficient in Data Modeling Techniques using Star Schema, Snowflake Schema, Fact and Dimension tables, RDBMS, Physical and Logical data modeling for Data Warehouse and Data Mart.

Experienced in Software Development Lifecycle (SDLC) using SCRUM, Agile methodologies. TECHNICAL SKILLS:

Big Data Ecosystem Hadoop Map Reduce, HDFS, Hive, HBase, Flume, Sqoop, Oozie, Kafka, Spark, and Zookeeper

Hadoop Distributions Apache Hadoop 2.x/1.x, Cloudera CDP, Hortonworks HDP Programming Languages Python, R, Scala, Java, SQL, HiveQL, PL/SQL, UNIX shell Scripting cloud Technologies AWS, S3, EC2, Redshift, IAM, Azure, Active Directory, Application Insights, Azure Monitoring, Azure Search, Data Factory, Key Vault and SQL Azure, Azure Devops, Azure Analysis services, Azure Synapse Analytics (DW), Azure Data Lake. Databases MySQL, Oracle, MS SQL SERVER

Version Control Git, Bitbucket

ETL/BI Informatica, SSIS, Tableau, Power BI

Operating System Mac OS, Windows 7/8/10, Unix, Linux, Ubuntu Methodologies UML, System Development Life Cycle (SDLC), Agile, Waterfall Model Work Experience:

Client: AT&T Jan 2023 – May 2023

Role: Azure Data Engineer

Responsibilities:

Developed and maintained end-to-end operations of ETL data pipelines, handling large data sets in Azure Data Factory.

Utilized SQL queries (DDL, DML) to implement indexes, triggers, views, stored procedures, functions, and packages.

Integrated on-premises (MYSQL) and cloud (Blob storage, Azure SQL DB) data using Azure Data Factory, applying transformations, and loading data back to Snowflake.

Orchestrated data pipelines using Data Factory, facilitating data flow into SQL databases.

Implemented Snowflake modeling techniques, including data warehousing, data cleansing, Slowly Changing Dimension handling, surrogate key assignment, and change data capture.

Applied an analytical approach to problem-solving, leveraging Azure Data Factory, data lake, and Azure Synapse for business problem resolution.

Developed ELT/ETL pipelines for data movement to and from Snowflake data store, utilizing Python and Snowflake Snow SQL.

Designed ETL transformations and validations using Spark-SQL/Spark Data Frames with Azure Databricks and Azure Data Factory.

Hands-on experience with Kafka, Spark Streaming for processing real-time streaming data in specific use cases.

Developed a data pipeline using Kafka, Spark, and Hive for data ingestion, transformation, and analysis.

Experienced in Agile methodologies, participating in daily stand-ups and internationally coordinated PI Planning sessions. Client: DXE Technologies Nov 2019 – Dec 2021

Role: Data Engineer

Responsibilities:

Hands-on experience in the complete project lifecycle (design, development, testing, and implementation) of Client-Server and Web applications.

Good experience in working with Azure cloud services and Apache Hadoop components, including Python, Spark, Scala, SQL, Hive, HDFS, Yarn, and MapReduce.

Proficient in Linux, VMWare, and container technologies within the Azure environment.

Hands-on experience with Azure data migration between database platforms, such as local SQL Servers to Azure RDS and Azure HDInsight.

Worked on HBase for conducting quick lookups, such as updates, inserts, and deletes within the Azure Hadoop environment.

Used Spark SQL to retrieve data from Hive and perform data processing using Spark API.

Converted Hive/SQL queries into Spark transformations for efficient data access and processing.

Experience in running MapReduce and Spark jobs over YARN within the Azure ecosystem.

Built scalable distributed data solutions using Azure HDInsight for processing large sets of structured and unstructured data.

Created batch and streaming pipelines in Azure Data Factory (ADF) to extract, transform, and load data from relational sources into Azure Data Lake Storage (ADLS Gen2).

Education Qualifications:

University of Central Missouri

Master’s in computer science

Sreenidhi Institute of science and technology,

Hyderabad, India

Bachelor of Technology in Computer science

Contact this candidate