Data Engineer - Python, Hadoop, Spark, Azure & AWS Experts

Nov **** - Jan ****

March **** - July ****

October 2022 - Feb 2023

Rohit Shrivastava

Shivpuram, Sisodia colony Guna (M.P)

913******* *****.***************@*****.***

https://www.linkedin.com/in/rohit-shrivastava-40b784258 Objective

Aspiring to leverage my 1.1 years of experience as a Data Engineer to design and optimize scalable data pipelines using Python, Hadoop, Hive, and Spark. Seeking an opportunity to contribute my expertise in Azure (ADLS Gen2, ADF, Synapse, Azure SQL) and AWS (S3, Glue, Athena, Quicksight, Redshift, Databricks) to build efficient ETL solutions, improve data quality, and enhance business intelligence capabilities. Experience

Ascendion Digital Solutions Pvt Ltd

Data Engineer

Muvi Entertainment pvt Ltd

Bigdata Engineer

Fourty Seven Billions Information Technologies Pvt Ltd Big data developer

Projects

Project 1: Border and Cie.

Domain worked on : Banking Domain

Technologies used : Hadoop, Hive, SQL, HBase, Sqoop, Cloudera. Roles and Responsibilities:

Take responsibility for Hadoop development and implementation Writing the script files for processing data and loading to HDFS. Loading files to HDFS and writing Hive queries to process data Involved in partitioning of Hive tables. Creating Hive tables to store the processed data in a tables. Setup Hive with MySQL as a remote Meta store.

Moved all the log/text files generated by various products into HDFS location. Project 2 : Data transformation to Data lake.

Domain worked on : Retail domain

Technologies used : Spark, Hive, SQL, HBase

Roles and Responsibilities :

Successfully completed POC on pipeline.

Responsible to merge data coming from different sources and loaded into HDFS. Supported code/design analysis, strategy development and project planning. Used Spark- SQL to process the data and to run the Spark engine. Combined data from MySQL and file source and applied various transformations. Creation of PySpark jobs.

Project 3 : Create 360 view for US Healthcare Industry Overview: Developed a comprehensive 360 view of patient and healthcare data. Technologies: SQL, Spark, PySpark, AWS S3, AWS Glue, Athena Roles and Responsibilities:

Integrated data from multiple healthcare sources into a centralized data lake Processed and transformed large-scale healthcare data using Spark and Hive Created data pipelines to ensure accurate and real-time insights Implemented data validation and governance policies for compliance Optimized data retrieval using partitioning and indexing 2013

2016

2021

Skills

Programming Languages: Python Scala

Big Data & Distributed Systems: Hadoop Hive Sqoop HBase PySpark Databases & Data Warehousing: MySQL Redshift RDS Azure SQL Data Warehousing ETL Cloud Technologies: AWS: S3 Glue Athena Quicksight RDS Redshift Databricks. Azure: ADLS Gen2 ADF Synapse Azure SQL

Data Engineering & Analytics: Data Pipelines Data Processing Data Integration Data Validation Performance Optimization

Education

Davv University

BCA

56 %

Jiwaji University

Bachelor of Law

63%

Jiwaji University

MSc (computer science)

71%

Personal Information

Name : Rohit Shrivastava

Marital status: Married

Address: Shivpuram Sisodiya Colony, Guna(M.P)

Pin : 473001

Contact this candidate