Azure Data Engineer - Azure Data Pipelines & Warehousing

Professional summary:

• Azure Data engineer with 14+ years of experience in Data warehousing with exposure to Design, Development, Testing, Maintenance, and customer support environments on multiple domains.

• 7+ years of experience in Azure Cloud, Azure Data Factory, Azure DataLake Storage Gen 1, Azure DataLake Storage Gen 2, Azure Synapse Analytics, and Databricks.

• Experience in designing and implementation of cloud architecture on Microsoft Azure.

• Excellent knowledge on integrating Azure Data Factory with variety of data sources and processing the data using the pipelines, pipeline parameters, activities, activity parameters, manually/window based/event-based job scheduling.

• Hands-on experience in developing Logic App workflows for performing event-based data

movement, perform file operations on Data Lake, Blob Storage, SFTP/FTP Servers,

getting/manipulating data in Azure SQL Server.

• Implemented Azure Active Directory Service for authentication of Azure Data Factory.

• Developed PySpark code to read data from source and create Spark data frames.

• Created python notebooks in Databricks Workspace, configured the notebook to read data from Datasets, and then ran Spark SQL jobs on the data.

• Worked on Data Warehouse design, implementation, and support (SQL Server, Azure SQL DB, Azure SQL Data warehouse).

• Experience in creating database objects such as Tables, Constraints, Indexes, Views, Indexed Views, Stored Procedures, UDFs and Triggers on Microsoft SQL Server.

• Strong experience in writing & tuning complex SQL queries including joins, correlated sub queries and scalar sub queries.

• Experience & Involved in all phases of SDLC process – Requirement Gathering, Analysis, Design, Coding, Code reviews, Configuration control, QA & deployment.

• Experience in Agile/SCRUM methodology.

Academic Background:

Bachelor’s in engineering (RGPV University, Bhopal, India)

Technical Skills:

Azure Cloud Platform

Azure Data Factory, Azure DataLake Storage Gen2, BLOB Storage, Azure SQL DB, SQL server, Azure Synapse Analytics, Data bricks, Key Vault, Azure App Services, Logic Apps, Event Grid

Programming Languages

PySpark, Python,

Databases

Azure SQL Warehouse, Azure SQL DB,, Microsoft SQL Server, MySQL,

IDE and Tools

SSIS, MS-Project, GitHub, JIRA, SharePoint,

Methodologies

Agile Scrum, SDLC,

Professional Experience:

Parent organization: Jarvis Consulting Group (2025 - 2026)

Client: CIBC 2025-July to till date

Role: Lead/Sr. Data Engineer

Location: Toronto

Responsibilities:

• Working as a Lead Engineer to design and implement scalable ETL/ELT pipelines using Azure Data Factory (ADF), integrating with Azure Data Lake Storage (ADLS Gen2) and Azure SQL Database to support enterprise data workloads.

• Designed and optimized data ingestion frameworks to process high-volume structured financial datasets with strong focus on performance, reliability, and fault tolerance.

• Developed complex transformation and reconciliation logic using Stored Procedures (T-SQL) and Databricks (PySpark/Spark SQL) for large-scale data processing.

• Leveraged Azure Synapse for data warehousing, dimensional modeling, and performance tuning to support downstream reporting and analytics.

• Developed and managed Delta Live Tables (DLT) pipelines to enable declarative ETL, built-in data quality checks, and simplified pipeline maintenance for streaming and batch data.

• Implemented Unity Catalog for centralized data governance, including fine-grained access control, data lineage tracking, and secure multi-workspace data management.

• Utilized Databricks Asset Bundles to standardize deployment of notebooks, workflows, and configurations, enabling robust CI/CD and environment promotion strategies.

• Built reusable orchestration frameworks in ADF using metadata-driven pipeline design, dynamic parameters, Lookup, ForEach, and Stored Procedure activities.

• Developed and deployed Azure Functions to implement custom validation logic, API integrations, and event-driven processing beyond native ADF capabilities.

• Integrated Azure Logic Apps to automate notifications, monitoring workflows, and cross-system integrations.

• Implemented robust data validation, reconciliation, and exception handling mechanisms to ensure regulatory compliance and data integrity in a banking environment.

• Collaborated with enterprise architects, business stakeholders, and QA teams to define data mapping, transformation rules, and migration checkpoints.

• Optimized pipeline performance and reduced processing time through parallelization, partitioning strategies, and workload optimization in Synapse and Databricks.

Parent organization: Ontario Securities Commissions (OSC) (2022 - 2025)

Client: Ontario Securities Commissions 2022 Feb – till date

Role: Data Engineer

Location: Toronto

Responsibilities:

• Migrate data from on-prem to cloud by developing ADF pipelines to migrate data from various data sources like DB2, SFTP, FTP, HTTP.

• Create synapse notebooks in various languages like python, spark to implementing cloud logic.

• Built ADF dataflow to migrate the data.

• Built design overall infrastructure for business users in OSC by providing secure platform.

• Maintain and provide support for optimal pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.

• Performed data validation, reconciliation, and quality checks, ensuring data accuracy and consistency across pipelines.

• Collaborated with cross-functional teams to troubleshoot issues, manage risks, and deliver scalable data solutions aligned with business requirements.

• Contributed to cost optimization strategies, resource planning, and cloud governance across Azure services.

• Ensured documentation, best practices, and governance standards were followed across all data engineering solutions.

Parent organization: Cognizant Technology Solution (2011 - 2022)

Client: Johnson & Johnson 2018-Aug – Feb 2022

Role: Data Engineer

Location: Canada

Responsibilities:

• Designed and implemented scalable data ingestion and transformation pipelines using Azure Data Factory (ADF) and Azure Databricks for both batch and near real-time processing.

• Architected end-to-end data pipeline solutions integrating multiple data sources (relational, semi-structured, and unstructured) into Azure Data Lake and Delta Lake.

• Developed robust ETL/ELT frameworks using PySpark and Spark SQL in Databricks for large-scale data processing, ensuring high performance and fault tolerance.

• Built and optimized ADF pipelines and data flows using activities such as Copy, Mapping Data Flows, ForEach, and Databricks notebooks for efficient orchestration.

• Implemented Delta Lake architecture including data ingestion, upserts (MERGE), and efficient data management using COPY INTO and DataFrame APIs.

• Designed and managed streaming data pipelines using Azure Event Hubs integrated with Databricks for real-time data processing.

• Configured and optimized Databricks clusters, jobs, and autoscaling, ensuring cost efficiency and performance tuning.

• Automated workflows using ADF triggers (event-based, schedule, tumbling window) for reliable and timely data processing.

• Leveraged Azure Synapse Analytics (serverless and dedicated SQL pools) for data exploration, querying, and analytics on large datasets.

• Implemented secure data access using Azure Key Vault and Azure Active Directory (AAD) for secrets management and authentication.

• Configured integration runtimes (Azure & self-hosted) and built linked services to securely connect on-premises and cloud data sources.

• Developed and optimized complex SQL logic (views, stored procedures, and transformations) for data modeling and downstream consumption.

• Enabled CI/CD pipelines using Jenkins for automated deployment of Databricks notebooks and data pipelines across environments.

• Worked on Azure-SSIS Integration Runtime to migrate and execute legacy SSIS packages in the cloud.

Environment: Azure SQL Server, Azure Data Warehouse, Azure Storage, SSIS, Azure Data Lake, Azure Data Lake Analytics, Azure Data Factory, Logic Apps, Function Apps, Event Hubs, Event Grids, SQL Server, Visual Studio.

Client: Johnson & Johnson Nov 2016 - Aug 2018

Role: Data Archival subject matter expert

Location: USA

Responsibilities:

• Requirements gathering from client for structured and unstructured data and analyzing their requirements.

• Analyze tables their relationship creates schema and entities based on tables, mining them with ILM, export/import entities.

• Create source and target connection along with target folders creation.

• After defining roles retention policies and groups with entities, run the archival jobs.

• Documents creation for all development, QA, production environments and get the approval wherever required from business unit and other stockholders.

• Analyze archival data provided by business unit for unstructured data.

• Develop automatic data extraction utilities in Python. Create project specific utilities in Python

• Track all the logs, issues in JIRA & Confluence.

• Validation of data through HP-ALM Tool.

• Installed ILM data archive products (Data archive and FAS) in windows/Linux.

Client: Johnson & Johnson Jul 2014 - Nov 2016

Role: Data Archival subject matter expert

Location: India

• Created custom entities with Business rules for online archive and file archive

• Verify data in history database/file targets

• Restored online archive data using transaction restore and file restore

• Verify the file archive data using search file archive and browse data

• Applied legal hold, Retention, Tagging for file archive data

• Created users in SQL worksheet.

• Restored file archive data using transaction and cycle restore

• Ran standalone jobs create tables, create indexes, create archive folder, seamless access job for Oracle.

• Involved Troubleshooting Customer issues and worked with GCS team

• Worked on Code migration using Enterprise Data Manager.

Environment: Hubstor, Informatica-ILM, Metalogix, IRIS, confluence, HP-ALM

Certifications:

Microsoft Certified Azure Data Engineer

Client: Merck June 2011 – June 2014

Role: SharePoint developer and Administrator

Location: India

Responsibilities:

• Design and develop custom solutions using Microsoft SharePoint 2010.

• Build and customize SharePoint sites, lists, libraries, and content types.

• Verifying source sites before migration and target sites after migration.

• Report creation for published sites,

• Create sites through PowerShell scripts and through Metalogix Migration manager tool.

• Performed Pre-Execution checks before site has being migrated, verification of whole sites those has migrated, fixed issues from my end, coordinated to Issue resolution team for critical issues update Status in TSM4 according to flow of site migration, report generation for upcoming publishing sites.

Environment: SharePoint 2010

Contact this candidate