Data Engineer

Dileep Kumar

Data Engineer

*.***************@*****.*** 551-***-**** https://www.linkedin.com/in/dileep-kumar-153487212/

PROFESSIONAL SUMMARY

•Senior Data Engineer with 9+ years designing and owning multi-cloud, end-to-end ETL/ELT Platforms across Streaming, Banking and Insurance domains.

•Engineered SQL server based ETL pipelines using stored procedures, complex SQL logic, Indexing strategies and performance tuning.

•Hands on Experience working on multi-cloud data platforms across Azure Databricks and AWS Iceberg/Snowflake, building distributed ETL pipelines supporting streaming and Analytical workloads.

•Engineered Delta Lake Pipelines in Azure Databricks with watermarking, Schema Evolution, Late arrival data handling and MERGE-based SCD2 logic for consistent ingestion and high-volume Streaming and Batch workloads.

•Built ML Ops Pipelines using Databricks workflows, ML flows tracking Feature store integration enabling automated model retraining, lineage tracking and reproducible experiment management.

•Designed medallion architecture (Bronze, Silver, Gold) for ingestion, Transformation and enrichment and ML feature preparation, ensuring standardized data flow across analytical and ML teams.

•Implemented Unity catalog for centralized governance RBAC lineage and cross workspace access control across multi cloud datasets.

•Built modular high-performance stored procedures for large-scale data processing, including transaction control, schema evolution, error handling and audit framework.

•Hands-on Snowflake architect designed Star and Snowflake schemas, Kimball dimensional models, and SCD Type 1 & 2 implementations with Snow pipe-based ingestion.

•Reduced compute costs by 20–30% through clustering keys, materialized views, and CDC-based incremental loading strategies.

•Extensive AWS data integration experience, S3 data lakes, Glue, DMS (full load + CDC), Redshift, and MSK (Kafka), to build governed data lakes and streaming ingestion for high Volume workloads.

•Built python pipelines to process API ingestion, outbound feeds, file generation and layer Validation, normalize messy third-party data and delivering clean datasets to billing, CRM and analytical systems.

•Migrated on-perm ETL workloads to AWS/Azure using ADF, Glue, Lambda, step functions and S3, implementing lifecycle policies, cost optimized ingestion patterns for large-volume datasets.

•Built Fabric Data Factory pipeline orchestrating ingestion from Oracle, SQL server and REST APIs into Fabric Lakehouse Applying Schema validation, CDC logic and automated reconciliation.

•Implemented Fabric Real-Time Analytical (KQL DB) for streaming ingestion, anomaly detection and low-level latency querying of telemetry and operational datasets.

•Designed Snowflake ingestion layers using Stages, file formats, streams, Tasks and stored procedures, solving cross cloud ingestion using Snow pipe, schema evolution and automated backfilling for late data.

•Leveraged Fabric warehouse for ETL transformation using stored procedures, partitioning strategies and performance - optimized SQL workloads.

•Published enterprise Power BI reports to the server and configured gateway based automatic refresh cycle eliminating manual updates and real time availability for business teams.

•Published dashboards to the Power BI cloud service, enabling distributed teams to access governed, version-controlled reports without relying on local files or manual distribution.

•Implemented Git/Azure DevOps, Jenkins CI/CD pipelines to eliminate manual deployments, enforcing version control, code reviews and automated testing for SQL, Python and ETL components.

•Used GitHub Copilot to accelerate SQL tuning, Stored procedure refactoring, Python Scripting and documenting Business critical documentation.

•Delivery Module Lead experience led cross-functional teams through requirements gathering, code reviews, ETL best practices enforcement, and Source-to-Target mapping documentation.

•Converted fragile corn-based jobs into AutoSys JIL workloads with dependency chains, calendars eliminating manual restarts and stabilize cross system ETL execution.

TECHNICAL SKILLS

Cloud & Data Platform: AWS (S3, Glue, Lambda, Step Functions, DMS, MSK, MWAA, RDS, EC2, CloudWatch, IAM, Secrets Manager) Azure (Databricks, Data Factory, Synapse Analytics, Event Hubs, Blob Storage, Key Vault) Azure Fabric: OneLake, Warehouse, Lakehouse, Fabric Pipelines GCP (Cloud Storage, BigQuery)

Data Lakehouse & Warehousing: Amazon Redshift, Azure Synapse, Snowflake (Snowpipe Streaming, Streams & Tasks, Snowpark, Cortex AI, Dynamic Data Masking, Row-Level Security, Performance Optimization) Databricks (Delta Lake, Unity Catalog, Auto Loader, Delta Live Tables, MLflow, PySpark, Spark SQL) Apache Iceberg PostgreSQL SQL Server Oracle DynamoDB

ETL & ELT: dbt (Core & Cloud) Talend Cloud Talend Data Integration Informatica PowerCenter SSIS

Orchestration & DevOps: Apache Airflow (MWAA) Autosys GitHub Actions CI/CD Terraform (IaC) Kubernetes Docker Git

Streaming & Replication: Apache Kafka (MSK) Snowpipe Streaming AWS DMS (Full Load + CDC) Fivetran Airbyte Debezium Kafka Connect Azure Event Hubs Parquet Avro

Data Modeling: Medallion Architecture Star & Snowflake Schema Kimball Dimensional Modeling SCD Type 1 & 2

AI/ML & Feature Engineering: Snowflake Cortex AI Databricks MLflow Snowpark ML PySpark MLlib Delta Live Tables

Governance & Compliance: AWS Lake Formation Azure Purview AWS Glue Data Catalog Dynamic Data Masking Row-Level Security PII Classification Data Contracts PCI-DSS SOX HIPAA

Languages: SQL (Advanced) Python (Snowpark, PySpark, Snowpark ML) Scala (Spark) Oracle PL/SQL Bash.

PROFESSIONAL EXPERIENCE

Sr Data Engineer SiriusXM NJ (Remote) Sep 2023 – Present

•Engineered and owned multicloud data architecture across Azure Databricks + AWS Snowflake/Iceberg to process Audio streaming events and subscriber telemetry at 50M+ events/day.

•Built high-volume Delta Lake ingestion pipelines that stabilized nightly workloads by eliminating schema drift, fixing malformed records and implementing automated late-arrival reconciliation.

•Developed Azure Data Factory pipelines orchestrating multi-source ingestions (SQL Server, Oracle, Rest APIs) with parameterized triggers, dependency changing for reliable cross cloud movement into Delta Lake.

•Refactored legacy PySpark jobs into adaptive pipelines using AQE broadcast joins and optimized shuffle boundaries cutting cluster execution time by over 40%

•Used Azure Databricks Unity Catalog for centralized governance, lineage, and crossworkspace access control across multicloud datasets.

•Leveraged Azure Data Factory for orchestrating crosscloud data movement into Databricks, reducing manual integration overhead.

•Built Fabric warehouse stored-procedure ETL flow that replaced brittle SQL Server jobs, improving refresh reliability and reducing manual intervention.

•Migrated Talend-based ETL into AWS native ingestion pipeline integrating external API ingestion Lambda, Glue processing, and Step Functions driven orchestration for production reliability.

•Implemented AWS Lake Formation for centralized governance, PII tagging, crossaccount access, and finegrained S3 permissions.

•Delivered cloud data migration initiatives from Oracle and PostgreSQL using AWS DMS (full load + CDC) into S3 and Snowflake, ensuring data integrity with validation and schema drift checks.

•Built realtime ingestion pipelines using MSK/Kafka, Lambda triggers and Snow pipe, enabling analytics teams to consume enriched data with subminute latency.

•Optimized large-scale Snowflake workloads processing daily events, reducing query runtime by 35% and compute costs by 20% using clustering, materialized views, and search optimization.

•Built Snowflake ELT pipelines using Streams & Tasks for incremental processing, automating CDC ingestion and reducing pipeline latency from hours to minutes.

•Designed Snowflakeready data models (Star, Snowflake, SCD2) for subscriber analytics, adtech reporting, and AI feature stores, improving downstream query performance and BI adoption.

•Implemented enterprise governance in Snowflake using RowLevel Security, Dynamic Data Masking, RBAC, and secure views to protect PII across subscriber and adtech datasets

•Built dbtdriven Snowflake transformations for Goldlayer analytics, enforcing testing, documentation, and lineage across 20+ production pipelines.

•Delivered on the go dashboards for Product, Marketing and AdsWizz teams, enabling real time visibility into audience engagement and addelivery performance

•Integrated Snowflake Cortex AI for realtime sentiment scoring on listener feedback and call transcripts, delivering predictions directly into Goldlayer Snowflake tables.

•Implemented Power BI semantic modeling and incremental refresh for various fact calculations and efficient data visualization in Power BI.

•Orchestrated endtoend Snowflake workflows using Airflow (MWAA) with SLAbased alerting, improving pipeline reliability and reducing failure detection time 60%.

Data Engineer American Express AZ Aug 2022 - Aug 2023

•Led modernization of Legacy Yellowbrick + Informatica/Talend ETL with a fully Snowflakedriven ELT architecture using Talend Cloud and SQL pushdown.

•Built reliable ingestion pipelines using AWS DMS (full + CDC), Talend, and Airflow to deliver validated, schemaconsistent financial datasets from Oracle, SQL Server, S3.

•Built Sparkaccelerated Big Data pipelines using Talend + AWS EMR/S3 to convert massive financial datasets into partitioned Parquet, improving Snowflake load throughput and query performance.

•Designed Snowflake financial data models including SCD2 dimensions, reconciled fact tables, and lineagetracked transformations aligned to accounting, FP&A, and regulatory reporting needs.

•Developed reconciliation & closeprocess pipelines implementing balancing logic, variance checks, and financial statement alignment for monthend close and audit workflows.

•Developed Kafka to Snowflake realtime ingestion for payment authorization events using Snowpipe Streaming with idempotent deduplication and schema validation.

•Built Python metadatadriven ELT framework that autogenerated Snowflake task graphs from config files, standardizing transformation patterns and reducing new financial pipeline onboarding time by 30%.

•Implemented Snowflake governance with Dynamic Data Masking, RowLevel Security, RBAC, and secure views to protect PII and regulated financial data.

•Built Talenddriven ELT frameworks orchestrating cleansing, enrichment, and accounting rulebased transformations directly inside Snowflake.

•Monitored EDW triggered .bat jobs executing SQL Server stored Procedures, implementing log-parsing, error-pattern detection and automated escalation to stabilize nightly financial batch cycles.

•Led requirement conversations with Amex stakeholders to convert business KPIs into scalable MicroStrategy datasets and highavailability dashboards

•Integrated MicroStrategy with Snowflake and internal APIs to deliver unified reporting across Ads, Offers, Dining, and Commercial Services

•Delivered in Agile environment participating in sprint planning, standups, retros, and release cycles for financefocused data engineering programs

•Orchestrated end-to-end financial data workflows using Apache Airflow for cross-pipeline dependency management, automating ELT scheduling, error handling, and SLA-based alerting across Snowflake and AWS data flows.

•Improved financial reporting efficiency by optimizing Snowflake ELT pipelines and reconciliation logic, reducing monthend close data availability time by 25% for Finance and FP&A teams

Data Engineer The Hartford CT (Remote) Mar 2020 - Aug 2022

•Executed on-premises Oracle and SQL Server insurance pipeline migration into AWS-hosted Snowflake using Talend ETL, delivering row-count, checksum, and SCD Type 2 reconciliation frameworks to ensure 100% data fidelity.

•Designed a Snowflake-centric data platform on AWS, landing 2M+ daily insurance records from S3 using Talend ETL jobs (tMap, tS3Get, tSnowflakeOutput) into raw, staged, and curated layers, orchestrated end-to-end via Autosys job scheduling.

•Reduced Snowflake compute costs by 30% by replacing full table reloads with incremental CDC-based Talend pipelines, capturing changes from source systems using timestamp and primary key strategies.

•Built audit-ready Snowflake reconciliation framework comparing curated layers with legacy actuarial systems, automated discrepancy detection supporting quarterly reserve calculations.

•Improved BI reporting speed by 45% by designing Star and Snowflake schemas optimized for actuarial and claims analytics, collaborating with reporting teams to align data models to business requirements.

•Optimized Snowflake query performance for high-volume insurance datasets by implementing clustering keys, partition pruning, and warehouse auto-scaling policies, reducing average query runtime by 35%.

•Implemented HIPAA-compliant PII masking at the Talend extraction layer and Snowflake Dynamic Data Masking, RBAC, region-based RLS, and SOX reconciliation controls to ensure end-to-end data protection.

•Built Snowflake data quality monitoring using reconciliation views and row-count checks, integrated with Autosys job-level alerting to ensure pipeline failures were detected and resolved before impacting actuarial BI and data science consumers.

ETL Developer SiriusXM NJ June 2017 - Feb 2020

•Engineered Talend pipelines to write Parquet-formatted subscriber data to AWS S3 using Hive-compatible partitioning, establishing the cloud landing zone foundation for downstream Snowflake ingestion.

•Introduced Kafka-based event streaming patterns for subscriber lifecycle events as part of early cloud readiness initiative, replacing file-based batch handoffs.

•Reduced data reload incidents by 50% by diagnosing and correcting SCD Type 2 logic on subscription lifecycle tables.

•Migrated 30+ Informatica PowerCenter mappings to Talend, processing 5M+ daily subscriber records with standardized logging and retry logic, reducing ETL maintenance overhead by 35% and improving pipeline observability.

•Built Oracle PL/SQL procedures and views for subscription reporting; maintained SSIS and SSRS during platform transition.

EDUCATION

Master of Science, Information Assurance / Information Technology 2017

Wilmington University New Castle, Delaware

Bachelor of Science, Mechanical Engineering 2015

JNTU Hyderabad, India

CERTIFICATIONS

SnowPro Core Certification — Snowflake 2026

Contact this candidate