Azure Data Engineer with Cloud Data Solutions Expertise

Professional summary:

** ***** ** ********** ** Data warehousing with exposure to Design, Development, Testing, Maintenance, and customer support environments on multiple domains.

5+ years of experience in Azure Cloud, Azure Data Factory, Azure DataLake Storage Gen 1, Azure DataLake Storage Gen 2, Azure Synapse Analytics, and Databricks.

Experience in designing and implementation of cloud architecture on Microsoft Azure.

Excellent knowledge on integrating Azure Data Factory with variety of data sources and processing the data using the pipelines, pipeline parameters, activities, activity parameters, manually/window based/event-based job scheduling.

Hands-on experience in developing Logic App workflows for performing event-based data

movement, perform file operations on Data Lake, Blob Storage, SFTP/FTP Servers,

getting/manipulating data in Azure SQL Server.

Implemented Azure Active Directory Service for authentication of Azure Data Factory.

Developed PySpark code to read data from source and create Spark data frames.

Created python notebooks in Databricks Workspace, configured the notebook to read data from Datasets, and then ran Spark SQL jobs on the data.

Worked on Data Warehouse design, implementation, and support (SQL Server, Azure SQL DB, Azure SQL Data warehouse).

Experience in creating database objects such as Tables, Constraints, Indexes, Views, Indexed Views, Stored Procedures, UDFs and Triggers on Microsoft SQL Server.

Strong experience in writing & tuning complex SQL queries including joins, correlated sub queries and scalar sub queries.

Experience & Involved in all phases of SDLC process – Requirement Gathering, Analysis, Design, Coding, Code reviews, Configuration control, QA & deployment.

Experience in Agile/SCRUM methodology.

Academic Background:

Bachelor’s in engineering (RGPV University, Bhopal, India)

Technical Skills:

Azure Cloud Platform

Azure Data Factory, Azure DataLake Storage Gen2, BLOB Storage, Azure SQL DB, SQL server, Azure Synapse Analytics, Data bricks, Key Vault, Azure App Services, Logic Apps, Event Grid

Programming Languages

PySpark, Python,

Databases

Azure SQL Warehouse, Azure SQL DB,, Microsoft SQL Server, MySQL,

IDE and Tools

SSIS, MS-Project, GitHub, JIRA, SharePoint,

Methodologies

Agile Scrum, SDLC,

Professional Experience:

Client: Bank 2025-July to till date

Role: Sr. Data Engineer

Responsibilities:

Implemented scalable ETL/ELT pipelines using Azure Data Factory (ADF) integrated with Azure Data Lake Storage (ADLS Gen2), Azure SQL Database.

Designed and optimized data ingestion frameworks to process high-volume structured financial datasets with strong focus on performance, reliability, and fault tolerance.

Developed complex transformation and reconciliation logic using Stored Procedures (T-SQL) and Databricks (PySpark/Spark SQL) for large-scale data processing.

Leveraged Azure Synapse for data warehousing, dimensional modeling, and performance tuning to support downstream reporting and analytics.

Built reusable orchestration frameworks in ADF using metadata-driven pipeline design, dynamic parameters, Lookup, ForEach, and Stored Procedure activities.

Developed and deployed Azure Functions to implement custom validation logic, API integrations, and event-driven processing beyond native ADF capabilities.

Integrated Azure Logic Apps to automate notifications, monitoring workflows, and cross-system integrations.

Implemented robust data validation, reconciliation, and exception handling mechanisms to ensure regulatory compliance and data integrity in a banking environment.

Collaborated with enterprise architects, business stakeholders, and QA teams to define data mapping, transformation rules, and migration checkpoints.

Optimized pipeline performance and reduced processing time through parallelization, partitioning strategies, and workload optimization in Synapse and Databricks.

Client: Ontario Securities Commission (OSC) 2023-feb to 2025- July

Role: Data Engineer

Responsibilities:

Migrate data from on-prem to cloud by developing ADF pipelines to migrate data from various data sources like DB2, SFTP, FTP, HTTP.

Create synapse notebooks in various languages like python, spark to implementing cloud logic.

Built ADF dataflow to migrate the data.

Built design overall infrastructure for business users in OSC by providing secure platform.

Maintain and provide support for optimal pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.

Client: Cognizant Technology Solution 2011-June to 2023-Jan

Role: Data Engineer

Responsibilities:

Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.

Design and developed Batch processing and real-time processing solutions using ADF, Databricks clusters.

Created numerous pipelines in Azure using Azure Data Factory to get the data from disparate source systems by using different Azure Activities like Move &Transform, Copy, filter, for each, Databricks etc.

Maintain and provide support for optimal pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.

Created python notebooks in Databricks, configure the notebook to read data from an Azure Open Datasets, and then run a Spark SQL job on the data.

Developed PySpark code to create Spark DataFrames with source Data from Azure Open Datasets, and use SQL to query the data.

Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.

Created, provisioned different Databricks clusters, notebooks, jobs and autoscaling.

Performed data flow transformation using the data flow activity.

Created Azure synapse workspaces and copy data into primary storage account.

Analyzed data by data exploration with built in serverless SQL pools and dedicated SQL pools to browse the contents of the files directly.

Integrated pipelines and activities using Synapse Studio by creating, scheduling and monitoring pipelines.

Configure and manage Azure key vaults to securely store keys, passwords, certificates, and other secrets.

Implemented Azure, self-hosted integration runtime in ADF.

Created Linked services to connect the external resources to ADF.

Working with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.

Configured Event-based triggers to run ADF pipelines in response to an event (storage/custom events).

Create, read, write, overwrite and append DeltaLake tables (delta format) by creating dataframes using Azure Databricks DataFrameReader.

Ingested data into DeltaLake tables by COPY INTO SQL command that allows loading data from a file location into a Delta table.

Ensure the developed solutions are formally documented and signed off by business.

Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue identification, and management.

Configured and set up data ingestion system using Azure Event Hubs and then connect it to Azure Databricks to process the messages/stream of data coming through.

Worked on the cost estimation, billing, and implementation of services on the cloud.

Environment: Azure Cloud, Azure Data Factory (ADF), Azure DataLake, BLOB Storage, SQL server, Windows remote desktop, Data bricks, Python, Azure SQL Server, Azure Data Warehouse.

Client: Johnson & Johnson Apr 2018 - Dec 2019

Role: Data Engineer

Responsibilities:

Designed and created optimal pipeline architecture on Azure platform.

Created pipelines in Azure using ADF to get the data from different source systems and transform the data by using many activities.

Worked on mapping data flows activities in the Azure data Factory.

Deploy databricks code into higher environment via Jenkin pipelines

Created Linked service to land the data from different sources to Azure Data Factory.

Worked on SQL Server Integration Services (SSIS) packages with Azure-SSIS integration runtime to run SSIS packages in ADF.

Implemented authentication mechanism using Azure Active Directory for data access and ADF.

Created different types of triggers to automate the pipeline in ADF.

Created, provisioned different Databricks clusters needed for batch and continuous streaming data processing and installed the required libraries for the clusters.

Created several Databricks Spark jobs with Pyspark to perform several tables to table operations.

Develop Azure SQL Data Warehouse SQL scripts with Polybase support and processing files stored in Azure Storage, Azure Data Lake.

Work on SQL Scripts, triggers, queries, packages to load data in SQL Server, and SQL Datawarehouse.

Environment: Azure SQL Server, Azure Data Warehouse, Azure Storage, SSIS, Azure Data Lake, Azure Data Lake Analytics, Azure Data Factory, Logic Apps, Function Apps, Event Hubs, Event Grids, SQL Server, Visual Studio.

Client: Johnson & Johnson Jul 2014 - Aug 2018

Role: Data Archival subject matter expert

Responsibilities:

Requirements gathering from client for structured and unstructured data and analyzing their requirements.

Analyze tables their relationship creates schema and entities based on tables, mining them with ILM, export/import entities.

Create source and target connection along with target folders creation.

After defining roles retention policies and groups with entities, run the archival jobs.

Documents creation for all development, QA, production environments and get the approval wherever required from business unit and other stockholders.

Analyze archival data provided by business unit for unstructured data.

Develop automatic data extraction utilities in Python. Create project specific utilities in Python

Track all the logs, issues in JIRA & Confluence.

Validation of data through HP-ALM Tool.

Installed ILM data archive products (Data archive and FAS) in windows/Linux.

Created custom entities with Business rules for online archive and file archive

Verify data in history database/file targets

Restored online archive data using transaction restore and file restore

Verify the file archive data using search file archive and browse data

Applied legal hold, Retention, Tagging for file archive data

Created users in SQL worksheet.

Restored file archive data using transaction and cycle restore

Ran standalone jobs create tables, create indexes, create archive folder, seamless access job for Oracle.

Involved Troubleshooting Customer issues and worked with GCS team

Worked on Code migration using Enterprise Data Manager.

Environment: Hubstor, Informatica-ILM, Metalogix, IRIS, confluence, HP-ALM

Certifications:

Microsoft Certified Azure Data Engineer

Client: Merck Mar 2011 – Feb 2013

Role: SharePoint developer and Administrator

Responsibilities:

Verifying source sites before migration and target sites after migration.

Report creation for published sites,

Create sites through PowerShell scripts and through Metalogix Migration manager tool.

Performed Pre-Execution checks before site has being migrated, verification of whole sites those has migrated, fixed issues from my end, coordinated to Issue resolution team for critical issues update Status in TSM4 according to flow of site migration, report generation for upcoming publishing sites.

Environment: SharePoint 2010

Contact this candidate