Data Engineer
Resume:
**********@*****.***
Nishant Agarwal
Data Engineer
Highly analytical and accomplished professional with 8+ years of proven expertise in creating, executing, and managing data solutions on the AWS and AZURE platform.
Proficient in data ingestion, transformation, and storage using AWS services, such as Apache Spark, Athena, Databricks, GLUE, Delta Tables, Data Pipelines, Apache Kafka, Kinesis AWS Code Commit and AZURE service such as Azure Databricks, Hive, Azure ADF, Azure EventHub, and Azure Stream Analytics. Skilled in code quality checks, root cause identification, and proactive operations management.
Technical Proficiencies
Programming Language:
Pyspark, Python, SQL, PL/SQL, R
AWS Data Engineering:
Apache Spark, Databricks, Athena, AWS GLUE, S3, Delta Lake, Delta Tables, Data Pipelines, Apache Kafka, AWS Kinesis.
AZURE Data Engineering:
Azure Databricks, Azure ADF, ADLS, Azure EventHub, Azure Stream Analytics.
Databases:
Teradata, Microsoft SQL Server, AWS Redshift, Oracle, Azure Synapse
ETL Technologies:
Informatica PowerCenter, Big Data Management, Data Quality
Data Analytics & Reporting:
Databricks SQL, Tableau
Bigdata Technologies
Hadoop, Hive, Apache Spark, Apache Kafka, API Integration
Data Warehousing:
Data modeling, ETL/ELT processes, data integration, data quality, dimensional modeling, OLAP, OLTP, MPP data warehouse, BI/DW architecture and concepts
Cloud Technologies (Git)
Azure DevOps, AWS DevOps (CodeCommit) – CI/CD Pipeline
Project Methodologies
Agile and Waterfall models
Data Analysis and Algorithms
Probability, Statistical methods/modeling, Hypothesis Testing, Linear/Logistic Regression, Optimization, Simulation, Supervised/Unsupervised machine learning, Clustering, Classification, Decision Tree, Random Forest
Education & Credentials
Master of Science in Business Analytics (Expected in 2023)
University of Cincinnati, Carl H. Lindner College of Business, Cincinnati, OH
Bachelor of Science & Technology in Software Engineering, 2013
SRM University, SRM Institute of Science and Technology Chennai, IN
Professional Experience
Marsh Mumbai, Maharashtra, IN 2021 to 2022
Module Lead
Created Azure data lake solution and utilized data cleaning procedures to enhance data quality. Devised design solution for capturing EventHub events and seamlessly transferring them to Azure Synapse through utilization of Azure Stream Analytics. Developed reusable PySpark scripts in Databricks to streamline data processing and integrated an Azure Data Factory pipeline to orchestrate seamless data flow into Azure Synapse SQL pool, effectively meeting ETL solution requirements. Provided regular updates to stakeholders on project progress and communicated risks on a weekly and monthly basis.
●Efficiently onboarded new team members within a two-week timeframe, while facilitating effective knowledge transfer on azure technologies to enhance team capabilities.
●Led team of seven members to successfully deliver project requirements within the specified timeline.
●Received an award for skillfully managing and executing a show with only 50% of the desired team capacity.
●Reduced database load by over 26% by changing table design structures and optimizing the queries.
Deloitte USI Mumbai, Maharashtra, IN 2018 to 2021
Consultant
Utilized Informatica BDM to develop reusable design framework for mass ingestion, requirements gathering, reviews, and AWS integration. Designed structured data ingestion into AWS S3, and unstructured data using Apache Kafka and Informatica Big Data. Completed knowledge transfer and project handover from external team. Implemented Audit logging framework with BDM and Unix script.
●Achieved a remarkable 90% increase in data accuracy and efficiency by implementing automation techniques to streamline manual data reconciliation processes across various data sources.
●Streamlined range of reusable generic BDM data flows, leading to substantial time savings of up to 50% in development endeavors.
●Played a key role in designing and implementing Fast Transfer Load (FTL) functionality across various workflow reporting modules.
●Reduced development time by 70% through automated parameter file generation framework.
●Recognized as project's technology stack SME, driving independent module execution.
Accenture Technologies Mumbai, Maharashtra, IN 2016 to 2018
Application Developer Analyst
Generated comprehensive data analysis and technical documentation encompassing source-to-target mappings, ensuring clear understanding, and seamless execution of data integration processes. Designed and developed robust workflows and mappings to efficiently ingest data from diverse sources, including Hive, flat files, MS SQL Server, and Oracle databases. Processed the data to facilitate seamless reporting capabilities.
●Implemented automated ETL (Extract, Transform, and Load) operations to simplify data manipulation processes and reduce time requirements by up to 40%.
●Resolved source data problems and redesigned transformation rules, improving data quality and accuracy.
Wipro & Cognizant Technologies Chennai, Tamil Nadu, IN 2014 to 2016
Project Engineer / Software Engineer
Utilized ETL tools as well as programming and scripting languages to develop, test, integrate, and deploy ETL processes, enabling efficient data extraction, transformation, and loading operations. Evaluated assigned tickets and provided estimated resolution times, effectively managing expectations. Promptly acknowledged ticket incidents and service requests in alignment with agreed-upon Service Level Agreements (SLAs).
●Executed data quality rules and effectively addressed data quality issues by 30% through the implementation of an exception-handling framework.
●Delivered L1 and L2 support, successfully reducing ticket counts by 20% through comprehensive redesign of the ETL process.
●Utilized Informatica PowerCenter and data quality methodologies to meet project requirements.
Awards & Honors
Catalyst of the Month, Marsh, 2021
Outstanding Performance Award, Deloitte, 2021
2 Applause Awards, Deloitte, 2019 2020
Contact this candidate