Data Engineer
Resume:
: ******************@*****.***
Data Engineer
Having 8+years of total IT experience in software development which includes 4 years as Data Engineer, 3years as ETL Developer and 1year as BI Developer.
Involved in the development of ETL process for extracting the data from heterogeneousdata sources, transforming the data,and loading the data into cloud systems using Azure Data Factory, Azure Data Bricks.
Experienced in Ingesting data to Azure Services like (ADLS Gen2, Delta Tables, Azure SQL DB) and processing data in data bricks.
Experienced in writing and unit testing PySpark code for transforming the data in data bricks.
Strong Knowledge on architecture and components of Spark, and efficient in working with Spark Core, Spark SQL.
Good Knowledge on Azure Databricks like provisioning spark cluster, creating Notebooks,
creating & passing parameters to the notebooks.
Experienced in building data pipelines for migrating the data from on-premises to ADLS gen2 andAzure SQL DBusing Azure Data Factory.
Integrated Azure Services like Azure Key Vault, Azure Blob Storage, Logic Apps, Azure Active Directorywith data bricks.
Knowledge of how to build Spark applications that leverage Spark-SQL in Databricks to extract, process, and aggregate data from a range of file types in order to analyze and alter the data in order to obtain insights into user behavior.
Good experience in implementing Full load and Incremental loading from on premises to cloud
using ADF.
ExperiencedinbuildingvisualizationreportsanddashboardsusingPower BI.
Created Talend jobs to migrate data from heterogeneous sources such as Oracle,
MSSQL Server and Flat Files to different types of target databases using various components.
Experience developing, supporting, and maintaining ETL (Extract, Transform, and Load) processes using Talend Integration Suite.
Experienced in implementingSCDType-2,Type-1Dimensionaltablesandfacts using Talend.
ExperiencedindevelopingVisualizationreportsand dashboardsusingTableauandpublishingthosereports to end users through Tableau Server.
Good exposure onData WarehousingconceptslikeStar Schema,SnowflakeSchema,
Dimension and Fact Tables.
Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Pyspark jobs.
Proficient knowledge in Designing and implementing data structures and commonly used data business intelligence tools for data analysis.
Proven experience in creating database objects like Tables, Views, and Writing Stored Procedures.
GeneratedSQL queries usingnecessaryjoin conditionsand usedindexesefficientlyfor good
performance of queries.
Managed various data pipelines using various Azure Devops methodologies.
Good knowledge in creating release pipelines in Azure Devops for moving data pipelines from one environment to another environment.
Experienced in building data flow pipelines for loading data from on premises systems to IBM Maximo using cloud integration tool Snap logic.
Cloud Technologies Microsoft Azure: ADF, ADLS & Azure Data Bricks, Azure Blob Storage, Azure Logic Apps, Azure Key Vault
Databases SQL Server 2012/2014/2016, Oracle 11g
ETL Tools Talend 6.5 & 7.2,Snap logic
BI Tools Tableau, Power BI
Programming Languages SQL, T-SQL
Database Tools SSMS, SQL developer, Azure Data Studio
Version Control Tool Git,Nexus
Operating Systems Windows 2007/10
DP-900 Azure DataFundamentals.
DA-100AnalyzingDatawithMicrosoftPowerBI.
Worked as Associate2 at Price Water House Coopers Sdc from Jul ‘21 to Mar ‘23.
Worked as Systems Engineer at Tata Consultancy Services Ltdfrom Jan ‘15 to July
’21.
Project Handlings:
Project-5
Role : Data Engineer
Client : NJR WAM, US
Duration : Aug ’21 to Mar ‘23
Technology : AzureDatabricks,Azure Data Factory,DataLakegen2,AzureDB, PySpark, Power BI
Responsibilities:
Analyzing the functional requirement documents and Mapping documents from Business.
Utilized Databricks, Spark to create notebooks for data extraction, analyzing and transforming the data according to the business requirements.
Developed Spark applications using Spark and Spark-SQL from multiple file formats.
Ability to apply the spark Data Frame API to complete data manipulation within spark session.
Written complex Pyspark code to make the SQL code simple for joins, sub queries and correlated sub queries.
Involved in loading and transforming large sets of Structured, Semi-Structured data and analyzed them by running Pyspark jobs.
Implemented SCD1 to load the data into Delta tables from ADLS Gen2.
Implemented performance optimization techniques in the notebooks.
Created pipelines to run the notebooks using ADF on a scheduled basis.
Involved in monitoring and schedulingthe pipelinesusingTriggersinAzureDataFactory.
Implemented changes in the existing pipelines and fixed bugs while debugging.
Performed Unit, System Integration Testing.
Identified defects and tracked them till closure using JIRA.
Project-4
Role : Data Engineer
Client : Kaiser Permanente, US
Duration : May ’19 to Jul ‘21
Technology : Azure Data factory,DataLakegen2,AzureSQLDB, MS SQL Server, SQL, Power BI
Responsibilities:
Plan,designandimplementapplicationAzureobjectssuchasPipelinesandSQLDatabase
objects such as stored procedures and views.
ExtensivelyworkedindataExtraction,TransformationandLoadingfromsourcetotarget
system using Azure Data Factory & Data Lake Gen2.
Implemented Incremental loading by connecting heterogenous sources using ADF.
StoringdataintoAzureDataLakeGen2indifferentfileformatslikeparquetetc.
Integrated ADF with email Azure logic apps for sending email notifications.
Createdreportsusingpowerbireportsandpublishedthemtopowerbiservice.
ResponsibleforCreatingTables,ViewsandStoredProceduresinAzureSQLdatabase
programming as required.
Developed multiple Power BI reports in Power BI desktop and published them.
Created DAX measures in Power BI.
Implemented performance tuning techniques for better execution of the reports.
OptimizedtheperformanceofquerieswithmodificationsinT-SQLqueries.
Project-3
Role : ETL Developer
Client : Invesco, US
Duration : Mar ’17 to Apr‘19
Technology : Talend DI 6.5, MS SQL andSQL
Responsibilities
Designing the ETL mappings and workflows to fetch data from multiple sources (.csv,.txt files) and also loading the data from these sources into relational tables or destination sources using Talend Open Studio.
Extensivelyuseddatabasecomponents,filecomponents,ftpcomponents,tmap,tfileList,
tSchemaComplainceCheck, tRunjob, tParallelize etc.
Workingknowledgeonthereusablecomponentslikecontexts,Globalvariables.
PreparemetadatainTalendIntegrationstudiorepository.
Importeddatafromvarioussources, transformedandloadedintoStagingArea.
Integratedjava codeinsideTalendstudiobyusingcomponentslike tJavaRow,tJava.
Project-2
Role : ETL Developer
Client : NSPI,US
Duration : Feb ’16 to Mar ‘17
Technology : Snap logic, IBM Maximo, dbeaver,SQL
Responsibilities:
Built data pipelines for extracting the data from source system, transforming the data and loading the data into target system.
Used various snaps like Rest snaps, Transform snaps, Database snaps.
Monitored pipelines and performed troubleshooting to fix the bugs.
Written JSON expressions and built complex mappings with json path expressions.
Implemented changes in the existing pipelines to increase the performance of the pipelines.
Built complex pipelines using various bulk snaps to ingest the millions of records.
Implemented framework to carry out SCD1 data ingestion.
Experienced in creating object structures, database configurations in IBM Maximo for loading the data.
Project-1
Role : BI Developer
Client : Citi Bank, US
Duration : Jan ’15 to Jan ’16.
Technology : Tableau,TableauServer,MSSQLServer
Responsibilities:
Experienced in building data models for creating the reports for end users.
Created dashboards using different visualizations like donut, butterfly, divergent bar charts and
cross tab charts.
Achieved conditional formatting for individual columns using dual axis.
Used actions like filter, URL and go to sheet while developing dashboards.
Used parameters, hierarchies, groups, combined fields, sets and cascade filters.
Optimized the reports with context filters and actions.
Generated complex dashboards using various actions like filtering, URL, Nested sorting andLOD's.
Provided Technical support to the users.
Bachelor of Technology in Electronics & Communications Engineering from JNTUH College of Engineering.
Contact this candidate