Senior Data Engineer - Cloud ETL & RAG

Kalyann

414-***-**** *********@*****.***

Senior Data Engineer

PROFESSIONAL SUMMARY

Senior Data Engineer with 13+ years of IT experience designing, developing, optimizing, and supporting scalable enterprise data platforms across CRM, healthcare, medical devices, retail, e-commerce, and healthcare payer domains.

Extensive experience building cloud-based data engineering solutions using Snowflake, Google Cloud Platform (GCP), AWS, BigQuery, Google Cloud Storage, Dataproc, and Amazon S3 to support enterprise analytics, reporting, and business intelligence.

Data Engineer with strong experience in cloud data platforms, data warehousing, ETL/ELT pipeline development, and large-scale data processing using Python, SQL, PySpark, Snowflake, Databricks, and cloud technologies.

Experienced in preparing, transforming, and optimizing structured and semi-structured data to support analytics, reporting, and AI-driven applications.

Knowledge of Generative AI, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), embeddings, and vector databases for building context-aware applications using enterprise data.

Familiar with developing RAG pipelines, including document ingestion, data chunking, embedding generation, semantic search, and retrieval of relevant information to provide contextual responses through LLMs.

Skilled in data quality, governance, secure data access, pipeline orchestration, and performance optimization, with an interest in integrating AI capabilities into enterprise data engineering solutions.

Strong expertise in end-to-end ETL/ELT development, batch and incremental processing, Change Data Capture (CDC), data integration, and workflow orchestration using Informatica Intelligent Cloud Services (IICS), SnapLogic, dbt, Python, PySpark, SQL, and Apache Airflow.

Proven experience developing scalable data pipelines, dimensional models, semantic data layers, data marts, and curated datasets that transform complex enterprise data into reliable, analytics-ready information.

Advanced knowledge of Snowflake development, SQL optimization, Spark performance tuning, BigQuery query optimization, data modeling, data quality, reconciliation, metadata management, data lineage, and cloud cost optimization.

Experienced in developing reusable transformation frameworks, automated data validation, incremental processing, production monitoring, and reliable recovery mechanisms for high-volume data workloads.

Strong understanding of Data Analytics and business intelligence data requirements, including KPI development, business metrics, dimensional analysis, historical trend analysis, aggregation, and standardized reporting datasets.

Hands-on programming experience with Python, PySpark, SQL, and Jinja, supporting data transformation, automation, validation, distributed processing, and analytical data preparation.

Experienced in implementing data security and governance practices, including Snowflake RBAC, dynamic data masking, row-access policies, least-privilege access, audit controls, and sensitive-data protection.

Proficient in Git, GitHub, Jenkins, CI/CD, automated testing, code reviews, Agile/Scrum, technical documentation, troubleshooting, and production support.

TECHNICAL SKILLS

Skill Category Technical Skills

Cloud Platforms & Data

Warehousing

Google Cloud Platform (GCP), Snowflake, BigQuery, AWS, Amazon S3, AWS Glue, Google Cloud Storage (GCS), Dataproc, Pub/Sub, Microsoft Azure, ADLS Gen2, IAM, Cloud Logging, Cloud Monitoring

Data Engineering & ETL/ELT

ETL/ELT, End-to-End Data Pipeline Development, Batch Processing, Incremental Loading, Change Data Capture (CDC), Metadata-Driven Ingestion, Informatica Cloud (IICS), SnapLogic, dbt, IBM DataStage, Matillion, Source-to-Target Mapping, API Ingestion, File Processing, Schema Evolution

Programming & Query

Languages

Advanced SQL, Python, PySpark, Jinja, Bash, Shell Scripting, Complex Joins, CTEs, Window Functions, Aggregations, Subqueries, MERGE Operations, Stored Procedures, JSON, CSV, REST APIs

Big Data & Distributed

Processing

Apache Spark, PySpark, Hive, HQL, Distributed Data Processing, Data Transformation, Partitioning, Caching, Join Optimization, Structured and Semi-Structured Data Processing, Spark Structured Streaming (Lab Experience)

Data Modeling & Architecture

Dimensional Modeling, Kimball Methodology, Star Schema, Snowflake Schema, Fact and Dimension Tables, Conformed Dimensions, Surrogate Keys, Data Marts, SCD Type 1/Type 2, Historical Data Management, Raw-to-Curated Architecture, Medallion Architecture Concepts, Data Vault 2.0, Semantic Models

Snowflake & dbt

Snowflake Tables, Views, Secure Views, External Stages, Snowpipe, Streams, Tasks, Stored Procedures, RBAC, Dynamic Data Masking, Row-Access Policies, Tagging, Query Profiles, dbt Models, Staging, Intermediate Models, Marts, Incremental Models, Snapshots, Macros, Sources, Tests, Documentation, Micro-Partition Pruning, Clustering, Warehouse Sizing, Compute Optimization

Data Analytics & Business

Intelligence

Analytics-Ready Datasets, Ad Hoc SQL Analysis, Data Profiling, KPI Calculations, Business Metrics, Aggregation, Trend Analysis, Dimensional Analysis, Historical Comparisons, Variance Investigation, Reporting Data Models, Customer, Sales, Product, Service, Inventory, Order, Healthcare Payer and Operational Analytics

Data Science & Analytical Data

Preparation

Python- and SQL-Based Data Preparation, Data Extraction, Cleansing, Feature-Ready Dataset Preparation, Aggregation, Transformation, Exploratory Data Preparation, Statistical Aggregation, Data Completeness, Consistency Checks, Anomaly Investigation, Data Quality Validation

Workflow Orchestration &

Streaming

Apache Airflow, Astronomer, DataStage Sequences, DAG Development, Scheduling, Dependency Management, Sensors, Retries, Logging, Alerting, Failure Recovery, Google Cloud Pub/Sub, Kafka Integration (Working Knowledge), Event-Time Processing, Watermarking, Late-Arriving Data Concepts, Pipeline Monitoring, Root-Cause Analysis Data Quality, Governance &

Security

Schema Validation, Schema-Drift Detection, Duplicate Detection, Referential-Integrity Checks, Source-to-Target Reconciliation, Control Totals, Row-Count Validation, Business- Rule Testing, Metadata Management, Data Lineage, Auditability, Data Classification, RBAC, Least-Privilege Access, Secure Views, Data Masking, Sensitive-Data Protection DevOps, Testing & CI/CD

Git, GitHub, Jenkins, CI/CD Pipelines, Branching Strategies, Pull Requests, Code Reviews, Automated Testing, Unit Testing, Integration Testing, Regression Testing, Deployment Validation, Release Management, Environment Promotion, Production Support, Troubleshooting, Technical Documentation

Databricks & Delta Lake

Databricks, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines (LDP), Structured Streaming, Unity Catalog, Change Data Feed (CDF), Z-Ordering, Liquid Clustering, Serverless Compute; familiarity with Kafka Integration, Late-Data Handling, Lakebase Synchronization, Zerobus, and LTAP Concepts (Certification/Lab Exposure) Methodologies & Professional

Skills

Agile/Scrum, Sprint Planning, Backlog Refinement, Technical Design Reviews, Requirements Analysis, Cross-Functional Collaboration, Stakeholder Communication, Problem-Solving, Production Troubleshooting, Continuous Improvement PROFESSIONAL EXPERIENCE

Salesforce — Senior Data Engineer March 2023 – Present San Francisco, CA, USA

Project Overview: Enterprise CRM and analytics data platform integrating customer, account, sales, service, product, and operational information into Snowflake for business analytics and reporting.

Designed and developed scalable ETL/ELT pipelines using Informatica Cloud (IICS), Snowflake, AWS S3, Python, PySpark, SQL, and dbt to integrate data from enterprise applications, APIs, relational databases, and files.

Implemented batch, incremental, and CDC ingestion patterns using timestamps, watermarks, business keys, and change- detection logic to process inserts, updates, and deletes efficiently.

Developed and optimized data ingestion and transformation pipelines to prepare structured and semi-structured enterprise data for analytics and AI-powered applications.

Supported the preparation of knowledge sources for Retrieval-Augmented Generation (RAG) by performing data cleansing, document processing, text chunking, metadata enrichment, and embedding preparation.

Explored the integration of Large Language Models (LLMs) with enterprise data sources to enable context-aware question answering and natural-language access to business information.

Worked with semantic search and vector database concepts to retrieve relevant document chunks and provide contextual information to LLM-based applications.

Applied data quality validation, metadata management, and access-control principles to improve the reliability, traceability, and security of data used in AI workflows.

Developed reusable dbt models, macros, snapshots, incremental transformations, and automated tests to convert raw source data into standardized, curated, and analytics-ready datasets.

Built complex SQL and PySpark transformations using joins, window functions, aggregations, MERGE operations, deduplication, cleansing, and enrichment to support enterprise data processing.

Designed and maintained Apache Airflow DAGs to orchestrate ingestion, transformation, data-quality checks, and downstream dependencies, implementing scheduling, retries, logging, and failure handling.

Designed dimensional models using Kimball methodology, including fact tables, dimension tables, conformed dimensions, and SCD Type 1/Type 2 to support consistent analytical reporting.

Developed semantic and presentation-layer datasets containing standardized business definitions, measures, hierarchies, and KPIs for customer engagement, sales pipeline, opportunity conversion, service performance, and product adoption.

Implemented data-quality and reconciliation frameworks using SQL, Python, and dbt to validate completeness, uniqueness, referential integrity, source-to-target record counts, and business rules.

Developed Snowflake objects including tables, views, secure views, stored procedures, Streams, Tasks, external stages, and Snowpipe integrations to support ingestion and transformation workloads.

Applied Snowflake security and governance controls, including RBAC, dynamic data masking, row-access policies, tagging, secure views, and least-privilege access for sensitive enterprise datasets.

Optimized SQL queries and Snowflake workloads through query-profile analysis, efficient joins, micro-partition pruning, warehouse sizing, and workload management.

Supported production deployments through GitHub version control, pull requests, code reviews, automated testing, CI/CD, release validation, incident troubleshooting, and Agile/Scrum collaboration. Dexcom — Senior Data Engineer April 2019 – February 2023 San Diego, CA, USA

Project Overview: Enterprise data integration platform consolidating customer experience, contact-center, commercial, medical-device, and operational information from SaaS applications and external feeds using SnapLogic, Google Cloud, Snowflake, and Airflow.

Designed and developed SnapLogic integration pipelines to ingest data from Qualtrics, Genesys, Veeva, IQVIA, authorized Dexcom Clarity data feeds, REST APIs, relational sources, and file-based systems.

Developed REST API integrations handling authentication, pagination, request/response processing, JSON parsing, schema mapping, and API error conditions.

Built reusable and parameterized ingestion frameworks supporting multiple source systems, environments, file locations, processing dates, and delivery schedules.

Implemented incremental ingestion using source timestamps, last-modified fields, watermarks, and business keys where supported by upstream applications.

Developed Snowflake SQL transformations using CTEs, joins, window functions, conditional logic, aggregations, and MERGE operations to integrate and standardize enterprise data.

Used Google Cloud services for storage, integration, processing, and movement of data between source systems and centralized analytical platforms.

Developed datasets combining customer experience, survey, contact-center, commercial, and operational information to support downstream reporting and analysis.

Implemented source-specific data-quality checks for missing records, duplicate files, invalid datatypes, schema changes, mandatory fields, and business-rule violations.

Built reconciliation and audit frameworks tracking batch identifiers, processing dates, source and target counts, execution status, rejected records, and error details.

Developed Apache Airflow DAGs to coordinate SnapLogic ingestion, Snowflake transformations, validation steps, and downstream dependencies, including retries, logging, scheduling, and failure notifications.

Implemented Snowflake access controls and supported data governance through RBAC, metadata documentation, data lineage, source-to-target mapping, and appropriate protection of sensitive healthcare information.

Managed GitHub-based development and deployments, investigated production issues involving API failures, schema drift, pipeline errors, and Snowflake loads, and collaborated with application owners, QA, architects, and analytics teams. Walmart — Data Engineer August 2014 – March 2019

Sunnyvale, CA, USA

Project Overview: Large-scale retail data engineering platform processing store transactions, products, orders, inventory, e-commerce, fulfillment, and supply-chain information using GCP, PySpark, Dataproc, Hive, Pub/Sub, and Airflow.

Designed and developed distributed data pipelines to process high-volume retail, transactional, inventory, product, order, and supply-chain datasets.

Built PySpark applications for data cleansing, filtering, joins, aggregations, deduplication, standardization, enrichment, and reconciliation across large datasets.

Used Google Cloud Storage as the data lake layer for raw, intermediate, and processed datasets and Google Cloud Dataproc to execute distributed Spark workloads.

Developed and optimized Hive SQL/HQL queries and Hive tables for structured data processing, aggregations, reconciliation, and analytical consumption.

Implemented Spark optimization techniques including partitioning, caching, broadcast joins, predicate pushdown, partition pruning, and efficient file sizing to improve workload performance.

Built batch ingestion pipelines for store transactions, product information, inventory movements, order activity, and operational data.

Developed near-real-time, event-driven ingestion patterns using Google Cloud Pub/Sub to support timely distribution of retail and operational events.

Created Apache Airflow DAGs using Astronomer to orchestrate ingestion, Dataproc execution, Spark transformations, data- quality validation, and downstream dependencies.

Implemented workflow scheduling, sensors, retries, logging, alerting, idempotent processing, restartability, and failure recovery to improve production reliability.

Developed curated analytical datasets supporting sales performance, inventory availability, product trends, order status, fulfillment efficiency, and supply-chain reporting.

Implemented data-quality checks, audit logging, source-to-target reconciliation, and exception handling to identify duplicate transactions, missing keys, schema inconsistencies, and incomplete data.

Used BigQuery for analytical querying and consumption of curated datasets; monitored GCP workloads using Cloud Logging and Cloud Monitoring and applied IAM-based access controls.

Troubleshot production Spark issues involving data skew, executor failures, memory constraints, excessive shuffling, small files, and upstream data-quality problems.

Maintained PySpark applications, SQL/Hive scripts, Airflow DAGs, and deployment artifacts in GitHub, supporting pull requests, code reviews, CI/CD, and controlled releases.

Collaborated with business teams, architects, application owners, and downstream analytics consumers to translate requirements into scalable data pipelines and reusable analytical datasets. Molina Health — Data Engineer March 2013 – July 2014 Long Beach, CA, USA

Project Overview: Healthcare payer data engineering initiative supporting the ingestion, transformation, validation, and analytical use of healthcare operational information on Google Cloud Platform and BigQuery.

Developed and supported data pipelines on Google Cloud Platform to ingest healthcare-related datasets from multiple source systems and file-based feeds.

Used BigQuery to store, transform, query, and organize healthcare datasets for reporting and analytical requirements.

Developed SQL transformations to cleanse, standardize, join, aggregate, and prepare data for downstream analytics and reporting.

Supported processing of healthcare payer datasets, including member, provider, eligibility, enrollment, and claims-related information, according to project requirements.

Created and maintained BigQuery tables, views, and analytical datasets to provide structured, reusable data for reporting teams.

Performed source-to-target validation, row-count comparisons, duplicate checks, null validation, datatype verification, and business-rule checks.

Optimized BigQuery queries through appropriate filtering, partitioning, clustering, and query-design practices to improve performance and manage processing costs.

Collaborated with senior engineers, analysts, and business stakeholders to clarify requirements, troubleshoot data issues, support deployments, and document data-processing workflows. EDUCATION

Stevens Institute of Technology — Hoboken, New Jersey, USA Master of Science in Computer Science

CGPA: 3.8

ADDITIONAL PROFESSIONAL COMPETENCIES

Data Engineering: Scalable pipeline architecture, data integration, distributed processing, orchestration, and cloud data warehousing.

Data Analytics: SQL analysis, KPI calculations, dimensional modeling, analytical datasets, data profiling, and reporting data preparation.

Data Science Support: Python-based data preparation, large-scale feature-ready dataset processing, aggregation, data validation, and analytical data quality.

Data Governance: Metadata, lineage, access control, auditability, sensitive-data protection, and data-quality management.

Engineering Delivery: Technical design, requirements analysis, peer reviews, CI/CD, production troubleshooting, and cross-functional collaboration.

Work Authorization and Employment: Available to discuss full-time U.S. employment opportunities and applicable work authorization during the recruitment process.

Contact this candidate