Azure Data Engineer

Subasshri Srinivasan

****************@*******.***

Project Engineer Wipro Technologies Ph.no: +91-900*******

Synopsis

Currently, working with Wipro Technologies and 3 years of experience in IT industry including Analysis, development and Testing and Implementation in Big data and Cloud platform.

Have cleared Certification Microsoft Azure Fundamentals (AZ-900).

Worked on Data Migration project using Azure Data factory, Azure Databricks and Azure SQL DB

Worked on creating azure pipelines for ingesting data from Azure Blob storage to Azure SQL DB using Azure Data Factory and automating hive scripts on Azure HDInsight cluster.

Worked on data migration from On-Premise databases to AWS S3 and Snowflake.

Worked on implementing SCD type 2 using Azure Databricks through Delta Tables.

Have worked on preparing architecture design diagram for migration proposals.

Trained and have hands-on experience in Apache Spark.

Attended trainings on Apache Spark for ADB core Technical training.

Technical Skill Set

Languages : Core Java, Python

IDEs : Eclipse, Jupyter

ETL : Ab-initio

Database : SQL Server

CI/CD : GitHub, Jenkins

Big Data Technologies : Apache Spark, HDFS, Hive

Azure Cloud Services : Azure Data Factory, Azure SQL DB, Azure Databricks

Operating systems : Linux (Ubuntu), Windows 10

Personal Skills

Strong Suit : Communication skills, Quick learner and Ability to perform as a team member

Qualities : Proactive, Confident and like taking up responsibilities.

Attitude : Positive, Friendly

Certifications

Microsoft certified Azure Fundamentals (AZ900) – April 2020

EMC certified Academic Associate in Cloud Infrastructure and Services – August 2015

Professional Experience

Project #3:

Team: Data Modernization

Role: Data Engineer

Environment and Technology: Azure Cloud Services (Azure Data factory, Azure SQL DB), Apache Spark

Description:

To modernize the traditional Mainframe process to Azure cloud services which is then consumed by Azure Applications

Responsibilities:

To Extract mainframe tables as flat files,store files in Azure blob storage and process the files using Apache Spark

To build tables in Azure SQL DB equivalent to DB2 tables from the delimited files through automated ADF pipelines

To develop .Net application from COBOL code and launch in Azure Functions App.

Function App to store output in Azure SQL DB, performance is maintained in Azure Dashboards and compared with Mainframe

Helped team in SQL DB and ADF pipeline optimizations and performance tunings

Designed the architecture diagram for the Data flow of the project

Project #2:

Team: Data transformation using Databricks

Role: Developer

Environment and Technology: Azure Databricks, Apache Spark.

Description:

The project aims to create various transformation to generate fact and dimension table of given customer service data from various sources and file formats which is to be updated with new history column/s based on the new changes made over the data records.

Roles & Responsibilities:

To build a Spark script to generate the Fact table and dimensional tables based on the requirement provided.

To generate the delta over the input data from the ADLS mount on the Databricks,which is been used to check with the existing table to remove duplicates and updated new and existing if any changes found in the columns as scripted the conditions based on SCD type 2 by using MERGE statement .

To store revised table data into another ADLS store mounted on the Databricks.

Prepared Technical documentation.

Project #1:

Team:Data Modernization

Role: Developer

Environment and Technology: AWS, Ab-initio, Unix shell scripting, Snowflake

Description:

Our client was moving away from Ab-initio DDE server to cloud server ECE. Project is to completely decommission the Ab-initio DDE server, Ab-initio ECE server and to do modernization of Ab-initio jobs to PySpark jobs (using Internal Spark framework)

Responsibilities:

To Analyze the jobs’ upstreams and downstreams using Ab-initio DDE server and design architecture diagrams

To write AWS Lambda functions to move files between S3 buckets and from S3 to Onelake and linked lambda with Cloudwatch, SNS/SQS to notify if lambda fails

Deployed AWS lambda using Bogie/One-Pipeline and on-boarded it to Jenkins

Created Webhook in GitHub to automatically trigger build in Jenkins

Academic Qualification

Qualifications

Institute

University

Duration

Percentage

B.E (Computer Science and Engineering)

PSNA College of Engineering and Technology, Dindigul

Anna University

2013-2017

70

Achievements

Received Best performer award and appreciations from offshore managers and the client.

PLACE: Chennai Subasshri Srinivasan

Contact this candidate