Azure Data Engineer

Professional summary:

Performing as Azure Tech Lead, driving cloud-first strategies by architecting scalable, secure, and cost-effective solutions on Microsoft Azure."

13+ years of experience in Data warehousing with exposure to Design, Development, Testing, Maintenance, and customer support environments on multiple domains.

5+ years of experience in Azure Cloud, Azure Data Factory, Azure DataLake Storage Gen 1, Azure DataLake Storage Gen 2, Azure Synapse Analytics, and Databricks.

Experience in designing and implementation of cloud architecture on Microsoft Azure.

Excellent knowledge on integrating Azure Data Factory with variety of data sources and processing the data using the pipelines, pipeline parameters, activities, activity parameters, manually/window based/event-based job scheduling.

Hands-on experience in developing Logic App workflows for performing event-based data

movement, perform file operations on Data Lake, Blob Storage, SFTP/FTP Servers,

getting/manipulating data in Azure SQL Server.

Implemented Azure Active Directory Service for authentication of Azure Data Factory.

Developed PySpark code to read data from source and create Spark dataframes.

Created python notebooks in Databricks Workspace, configured the notebook to read data from Datasets, and then ran Spark SQL jobs on the data.

Worked on Data Warehouse design, implementation, and support (SQL Server, Azure SQL DB, Azure SQL Data warehouse).

Experience in creating database objects such as Tables, Constraints, Indexes, Views, Indexed Views, Stored Procedures, UDFs and Triggers on Microsoft SQL Server.

Strong experience in writing & tuning complex SQL queries including joins, correlated sub queries and scalar sub queries.

Experience & Involved in all phases of SDLC process – Requirement Gathering, Analysis, Design, Coding, Code reviews, Configuration control, QA & deployment.

Experience in Agile/SCRUM methodology.

Academic Background:

Bachelor’s in engineering (RGPV University, Bhopal, India)

Technical Skills:

Azure Cloud Platform

Azure Data Factory, Azure DataLake Storage Gen2, BLOB Storage, Azure SQL DB, SQL server, Azure Synapse Analytics, Data bricks, Key Vault, Azure App Services, Logic Apps, Event Grid

Programming Languages

PySpark, Python,

Databases

Azure SQL Warehouse, Azure SQL DB,, Microsoft SQL Server, MySQL,

IDE and Tools

SSIS, MS-Project, GitHub, JIRA, SharePoint,

Methodologies

Agile Scrum, SDLC,

Professional Experience:

Client: Ontario Securities Commissions Dec 2022 - Till Date

Role: Data Engineer

Responsibilities:

Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.

Design and developed Batch processing and real-time processing solutions using ADF, Databricks clusters.

Created numerous pipelines in Azure using Azure Data Factory to get the data from disparate source systems by using different Azure Activities like Move &Transform, Copy, filter, for each, Databricks etc.

Maintain and provide support for optimal pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.

Created python notebooks in Databricks, configure the notebook to read data from an Azure Open Datasets, and then run a Spark SQL job on the data.

Developed PySpark code to create Spark DataFrames with source Data from Azure Open Datasets, and use SQL to query the data.

Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.

Created, provisioned different Databricks clusters, notebooks, jobs and autoscaling.

Performed data flow transformation using the data flow activity.

Created Azure synapse workspaces and copy data into primary storage account.

Analyzed data by data exploration with built in serverless SQL pools and dedicated SQL pools to browse the contents of the files directly.

Integrated pipelines and activities using Synapse Studio by creating, scheduling and monitoring pipelines.

Configure and manage Azure key vaults to securely store keys, passwords, certificates, and other secrets.

Implemented Azure, self-hosted integration runtime in ADF.

Created Linked services to connect the external resources to ADF.

Working with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.

Configured Event-based triggers to run ADF pipelines in response to an event (storage/custom events).

Create, read, write, overwrite and append DeltaLake tables (delta format) by creating dataframes using Azure Databricks DataFrameReader.

Ingested data into DeltaLake tables by COPY INTO SQL command that allows loading data from a file location into a Delta table.

Ensure the developed solutions are formally documented and signed off by business.

Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue identification, and management.

Configured and set up data ingestion system using Azure Event Hubs and then connect it to Azure Databricks to process the messages/stream of data coming through.

Worked on the cost estimation, billing, and implementation of services on the cloud.

Environment: Azure Cloud, Azure Data Factory (ADF), Azure DataLake, BLOB Storage, SQL server, Windows remote desktop, Data bricks, Python, Azure SQL Server, Azure Data Warehouse.

Client: Cognizant Technology Solution

June 2011 - Feb 2022

Role: Data Archival subject matter expert

Responsibilities:

Designed and implemented data pipelines on Azure for processing large-scale data.

Created ADF pipelines to extract, transform, and load (ETL) data from on-premises and cloud sources.

Developed SQL scripts and queries for data extraction and transformation in Azure SQL Data Warehouse.

Integrated Azure Active Directory (AAD) for secure authentication and access control in ADF.

Developed CI/CD pipelines for automated deployment of data pipelines using Azure DevOps.

Worked on data archiving solutions and optimized storage strategies for cost efficiency.

Implemented PolyBase and external tables in Azure Synapse to query external data sources.

Developed parameterized ADF pipelines for dynamic data movement across multiple environments

unAnalyze tables their relationship creates schema and entities based on tables, mining them with ILM, export/import entities.

Create source and target connection along with target folders creation.

After defining roles retention policies and groups with entities, run the archival jobs.

Documents creation for all development, QA, production environments and get the approval wherever required from business unit and other stockholders.

Analyze archival data provided by business unit for unstructured data.

Develop automatic data extraction utilities in Python. Create project specific utilities in Python

Track all the logs, issues in JIRA & Confluence.

Validation of data through HP-ALM Tool.

Installed ILM data archive products (Data archive and FAS) in windows/Linux.

Created custom entities with Business rules for online archive and file archive

Verify data in history database/file targets

Restored online archive data using transaction restore and file restore

Verify the file archive data using search file archive and browse data

Applied legal hold, Retention, Tagging for file archive data

Created users in SQL worksheet.

Restored file archive data using transaction and cycle restore

Ran standalone jobs create tables, create indexes, create archive folder, seamless access job for Oracle.

Involved Troubleshooting Customer issues and worked with GCS team

Worked on Code migration using Enterprise Data Manager.

Environment: Hubstor, Informatica-ILM, Metalogix, IRIS, confluence, HP-ALM

Certifications:

Microsoft Certified Azure Data Engineer

Client: Mar 2011 – Feb 2016

Role: SharePoint developer and Administrator

Responsibilities:

Verifying source sites before migration and target sites after migration.

Report creation for published sites,

Create sites through PowerShell scripts and through Metalogix Migration manager tool.

Performed Pre-Execution checks before site has being migrated, verification of whole sites those has migrated, fixed issues from my end, coordinated to Issue resolution team for critical issues update Status in TSM4 according to flow of site migration, report generation for upcoming publishing sites.

Environment: SharePoint 2010

Contact this candidate