Data Engineer Azure Devops

Lavanya Nag Kammila

Email: **************@*****.***

Mobile: +1-513-***-****

Data Engineer

PROFESSIONAL SUMMARY:

Results-driven Data Engineer with 3+ years of experience demonstrating analytical thinking and attention to detail in designing data solutions, working in Agile/Scrum teams. Strong problem-solving skills applied to complex data challenges.

Developed dynamic data pipelines integrating various systems, showcasing proficiency in connecting dots across applications and businesses to understand the E2E view, ensuring team success.

Created PySpark and Python scripts for data transformation, demonstrating strong PL/SQL skills in writing and analyzing complex queries and stored procedures, with innovative thinking.

Designed real-time streaming pipelines, effectively communicating across the organization to both technical and non-technical audiences, and identifying priorities effectively.

Automated deployments of data pipelines using Azure DevOps, ensuring CI/CD compliance, and demonstrating know-how in working in Agile/scrum teams for prioritization of work.

Engineered robust data quality frameworks, ensuring accuracy and reliability, while demonstrating expertise with Microsoft Office suite usage and willingness to ask questions.

Integrated third-party data sources using REST APIs, enhancing data availability, and demonstrating experience with Oracle Exadata or 10g and above, as a strong team player.

Built Power BI dashboards to support executive reporting, operational metrics, and user-driven self-service analytics initiatives, managing multiple projects simultaneously.

Monitored costs in Azure and GCP, establishing proactive alerts, and demonstrating know-how in effort and financials estimation, working well in a team environment.

Worked with analysts, architects, and QA teams to define data requirements, implement delta loads, and automate refresh schedules, able to influence and guide team for success.

TECHNICAL SKILLS:

Databases - Azure SQL DB, Oracle, MySQL, PostgreSQL, MongoDB, BigQuery, Oracle Exadata

Languages - SQL, Python, PySpark, Shell Scripting, PL/SQL

DevOps - Azure DevOps, Git, Jenkins, Terraform

Others - Microsoft Office Suite

PROFESSIONAL EXPERIENCE:

IPrime Info Solutions Inc December 2024 – Present

Data Engineer

Responsibilities:

Applied analytical thinking to engineer modular Azure Data Factory pipelines, ensuring attention to detail in parameterized linked services for dynamic data ingestion and transformation across diverse sources. This enabled scalable enterprise integration needs, demonstrating problem-solving skills and innovative thinking.

Developed high-performance PySpark jobs in Azure Databricks, showcasing strong PL/SQL skills through distributed transformations, complex joins, aggregations, and surrogate key logic over structured and semi-structured datasets for downstream analytics. This required proficiency with query tools.

Built and optimized Delta Lake storage in ADLS Gen2 with partitioning and schema evolution, enabling high-volume ACID-compliant data storage for real-time business use cases and advanced reporting, demonstrating attention to detail. This required analytical thinking.

Integrated Azure Key Vault secrets into ADF and Databricks workflows, securing credentials and connection strings, improving compliance with SOC2, HIPAA, and internal cloud governance frameworks, showcasing problem-solving skills. This required innovative thinking.

Implemented real-time streaming solutions using Event Hub and Spark Structured Streaming for ingesting and processing application logs, clickstream data, and IoT signals from distributed sources, demonstrating strong communication skills. This required attention to detail.

Created reusable ADF pipeline templates and data flows with custom expressions, making it easier to onboard new datasets and reduce code redundancy across cross-functional project teams, showcasing analytical thinking. This required problem-solving skills.

Defined and enforced data quality rules through Python-based validations and Data Flow activities, detecting schema violations and null value anomalies early in the ingestion lifecycle, demonstrating attention to detail. This required innovative thinking.

Established CI/CD pipelines in Azure DevOps for Databricks notebooks, ADF resources, and ARM templates, enabling version control, automated testing, and streamlined production deployments, showcasing problem-solving skills. This required analytical thinking.

Built comprehensive data lineage, glossary, and classification tagging using Azure Purview to support data governance, metadata traceability, and GDPR compliance for enterprise reporting, demonstrating strong communication skills. This required attention to detail.

Partnered with Power BI developers to ensure optimized star schema data models, establishing shared datasets and managing refresh scheduling through workspace integration with Synapse and ADF pipelines, showcasing analytical thinking. This required problem-solving skills.

Tata Consultancy Services May 2021 – July 2023

Data Engineer

Responsibilities:

Developed scalable data pipelines in Azure Data Factory for ingesting data from Oracle, SAP, and flat files, implementing scheduling, error handling, and custom logging using control flow and parameterized triggers, demonstrating analytical thinking. This required attention to detail.

Optimized Spark workloads on Azure Databricks by tuning memory configurations, repartitioning strategies, and caching logic, achieving a 30% reduction in overall pipeline execution time, showcasing problem-solving skills. This required innovative thinking.

Created end-to-end ELT flows with data landing in ADLS Gen2 and transforming into curated Delta tables consumed by Synapse for enterprise-wide dashboard reporting, demonstrating strong communication skills. This required attention to detail.

Translated legacy SSIS packages into modern ADF equivalents, rebuilding transformations, conditional splits, and lookups while maintaining exact data output logic and validation checks, showcasing analytical thinking. This required problem-solving skills.

Integrated data from Salesforce and ServiceNow into Azure SQL using REST API connectors and custom Python wrappers within Azure Functions to support CRM data migration, demonstrating attention to detail. This required innovative thinking.

Built modular PySpark notebooks with dbutils widgets for dynamic parameterization and executed multi-step transformation logic as reusable components in multi-tenant data platforms, showcasing problem-solving skills. This required analytical thinking.

Supported role-based access control by defining RBAC roles and linked service credential separation using Managed Identity and Azure Key Vault in ADF, demonstrating strong communication skills. This required attention to detail.

Created and maintained detailed mapping documents, SQL specs, and operational support manuals to facilitate handoffs and ease UAT and production deployments, showcasing analytical thinking. This required problem-solving skills.

Participated in peer code reviews, Spark job debugging, and performance optimization workshops, raising the overall engineering maturity within the data team, demonstrating strong communication skills. This required attention to detail.

Contributed to internal data engineering playbooks with templates for naming conventions, logging strategies, and cluster configuration recommendations for Databricks pipelines, showcasing problem-solving skills. This required analytical thinking.

Tata Consultancy Services January 2021 – May 2021

Data Analyst

Responsibilities:

Conducted in-depth data profiling and cleansing tasks using SQL and Excel to identify duplicates, nulls, and outliers before ingestion into reporting data marts and dashboards, demonstrating analytical thinking and attention to detail. This required problem-solving skills.

Designed stored procedures and SQL views to build foundational layers for Power BI reporting, covering sales, marketing, and financial datasets in SQL Server, showcasing strong PL/SQL skills and proficiency with query tools. This required innovative thinking.

Built interactive Power BI dashboards using DAX measures, drill-through filters, and KPI indicators, empowering stakeholders with actionable visual insights from curated datasets, demonstrating strong communication and presentation skills. This required attention to detail.

Created robust SQL scripts to extract daily operational data from staging tables, joining and aggregating datasets into business-ready reporting views, showcasing analytical thinking and attention to detail. This required problem-solving skills.

Documented transformation logic, mapping rules, and column-level metadata for knowledge management and smooth handover to ETL development teams, demonstrating strong communication skills and attention to detail. This required innovative thinking.

Collaborated with BI developers and QA analysts to validate output reports against expected values, uncovering mismatches and contributing to enhanced report accuracy, showcasing analytical thinking and attention to detail. This required problem-solving skills.

Worked with Excel-based macros and VBA scripts to automate daily reporting tasks, reducing manual efforts and minimizing repetitive errors in operational metrics, demonstrating proficiency with Microsoft Office suite usage. This required innovative thinking.

Provided ad hoc analysis support by querying sales and finance data on demand and visualizing trends for leadership decisions using Excel and Power BI, showcasing analytical thinking and attention to detail. This required problem-solving skills.

Participated in data migration exercises, validating row counts, data types, and integrity checks across source and target databases to ensure successful ETL pipeline testing, demonstrating strong communication skills. This required attention to detail.

Built basic Python scripts to read flat files, cleanse formatting issues, and insert clean data into staging tables, supporting data onboarding during project kickoffs, showcasing analytical thinking. This required problem-solving skills.

Certifications:

Microsoft Certified: Azure Fundamentals (AZ-900)

Educational Details:

Master of Science in Computer Science - University of Cincinnati

Bachelor of Technology in Electronics Communication - VelTech Dr. RR & Dr. SR College

Contact this candidate