Data Engineer

Shivani Raj

Email: *****************@*****.*** PH: 765-***-****

Senior Data Engineer

PROFESSIONAL SUMMARY

** ***** ** ********** ** Data Engineering, designing, developing, and deploying scalable enterprise data platforms, ETL/ELT pipelines, distributed data processing solutions, and cloud-based analytics architectures.

Strong hands-on experience with Google Cloud Platform (GCP), developing production-grade data pipelines using BigQuery, Cloud Storage (GCS), Dataproc, Dataflow, Pub/Sub, Cloud Composer/Airflow, and related GCP services.

Extensive experience developing scalable Apache Spark applications using Java, PySpark, and SQL for large-scale batch and distributed data processing workloads.

Strong Java development experience focused on data engineering, Spark applications, distributed processing, ETL frameworks, data transformations, exception handling, and production pipeline development.

Extensive experience designing and implementing Apache Airflow DAGs for enterprise data pipeline orchestration, scheduling, dependency management, retries, monitoring, alerting, and failure recovery.

Proven expertise developing high-volume batch and streaming data pipelines using Apache Spark, GCP Dataflow, Pub/Sub, Dataproc, and Airflow.

Strong experience designing end-to-end ETL/ELT workflows covering data ingestion, transformation, cleansing, validation, enrichment, aggregation, reconciliation, and downstream publishing.

Advanced SQL expertise with complex joins, CTEs, subqueries, window functions, aggregations, analytical transformations, incremental processing, data validation, and performance optimization.

Extensive BigQuery experience developing scalable analytical data models, SQL transformations, partitioned datasets, clustered tables, incremental pipelines, and high-performance analytical workloads.

Hands-on experience building GCS-based data ingestion frameworks for structured and semi-structured data, integrating cloud storage with Spark, Dataproc, BigQuery, Dataflow, and Airflow.

Strong experience optimizing Apache Spark jobs using partitioning, repartitioning, coalescing, caching, broadcast joins, predicate pushdown, shuffle optimization, parallel processing, and optimized file layouts.

Experienced in developing reliable and restartable data pipelines with comprehensive error handling, data-quality validation, audit logging, retry mechanisms, checkpointing, and operational monitoring.

Strong background working with Big Data technologies and distributed computing architectures to process high-volume healthcare, pharmaceutical, energy, retail, transactional, and operational datasets.

Extensive experience implementing data-quality frameworks covering completeness, uniqueness, accuracy, referential integrity, duplicate detection, schema validation, reconciliation, and business-rule validation.

Proven ability to troubleshoot production Airflow DAGs, Spark applications, Dataproc clusters, BigQuery workloads, Dataflow pipelines, Pub/Sub integrations, and GCP infrastructure issues.

Experienced in developing cloud migration and modernization solutions, transforming legacy ETL and data-processing workloads into scalable cloud-native architectures using GCP, Spark, Airflow, BigQuery, and GCS.

Strong experience implementing both batch and near-real-time data processing architectures, integrating event-driven ingestion with Pub/Sub, Dataflow, Spark, and downstream BigQuery platforms.

Extensive healthcare and pharmaceutical data engineering experience working with claims, eligibility, member, patient, provider, clinical, prescription, dispensing, medication, reimbursement, utilization, and operational datasets.

Strong understanding of distributed data processing, cloud data platforms, data modeling, ETL/ELT architecture, pipeline orchestration, data governance, performance optimization, and enterprise data engineering best practices.

Proven ability to collaborate with application developers, data architects, analysts, business stakeholders, and engineering teams to deliver scalable, reliable, high-performance data solutions aligned with enterprise business requirements.

PROFESSIONAL EXPERIENCE

Senior Data Engineer

MISO Energy, Remote Jan 2026 to Present

Responsibilities:

Designed and developed scalable GCP data engineering pipelines using Java, Apache Spark, PySpark, Airflow, BigQuery, Cloud Storage (GCS), Dataproc, Pub/Sub, and Dataflow to process high-volume energy, operational, and transactional datasets.

Developed Java and Spark-based batch processing applications on Dataproc, implementing distributed transformations, joins, aggregations, partitioning, and parallel processing for large-scale datasets.

Built and maintained Apache Airflow DAGs to orchestrate multi-stage ETL/ELT workflows across GCS, Dataproc, BigQuery, and downstream analytical platforms, including dependency management, retries, scheduling, and failure handling.

Engineered batch and streaming data pipelines using Spark, Pub/Sub, and Dataflow, processing operational events and near-real-time data feeds while maintaining scalability and pipeline reliability.

Developed complex SQL transformations in BigQuery for data cleansing, normalization, aggregation, dimensional modeling, reconciliation, and analytical reporting.

Implemented GCS-based data ingestion frameworks to land structured and semi-structured source data into cloud storage before distributed processing through Spark/Dataproc.

Optimized Spark workloads through partitioning, caching, broadcast joins, predicate pushdown, file-size optimization, and reduction of unnecessary shuffles, improving performance for high-volume processing jobs.

Developed reusable ETL/ELT frameworks using Java, Spark, SQL, and Airflow, incorporating incremental processing, error handling, data validation, audit logging, and restartable workflows.

Monitored and troubleshot production Airflow DAGs, Spark jobs, Dataproc clusters, and GCP pipeline failures, analyzing logs and execution metrics to resolve data-quality, performance, and infrastructure issues.

Supported AWS-to-GCP/cloud modernization and data migration initiatives, transforming legacy ETL processes into scalable cloud-native pipelines using GCS, BigQuery, Dataproc, Spark, Airflow, and SQL.

Senior Data Engineer

Humana Inc., Remote Dec 2023 to Dec 2025

Responsibilities:

Developed enterprise GCP data pipelines using Java, Apache Spark/PySpark, Airflow, BigQuery, GCS, Dataproc, Pub/Sub, and Dataflow to process healthcare claims, member, provider, clinical, and operational datasets.

Built distributed Spark ETL pipelines for high-volume healthcare data, implementing transformations, deduplication, joins, aggregations, schema standardization, and business-rule processing.

Created Apache Airflow DAGs to orchestrate ingestion, Spark processing, BigQuery transformations, data-quality validation, and downstream healthcare analytics workflows.

Developed complex BigQuery SQL using CTEs, window functions, nested structures, aggregations, and incremental processing logic for claims, eligibility, provider, utilization, and clinical datasets.

Implemented Java-based Spark applications for scalable data processing, leveraging distributed execution and parallel transformations to support large healthcare datasets.

Designed cloud ETL/ELT workflows using GCS Dataproc/Spark BigQuery, establishing standardized ingestion, transformation, validation, and publishing layers.

Integrated event-driven data processing using Pub/Sub and Dataflow, supporting ingestion of application and operational healthcare events into cloud analytical platforms.

Optimized Spark processing through partition pruning, broadcast joins, caching, repartitioning/coalescing, optimized file formats, and shuffle reduction for large-scale claims and member datasets.

Implemented data-quality controls for member eligibility, claims, provider, diagnosis, procedure, pharmacy, and utilization data, including null checks, duplicate detection, referential validation, and reconciliation.

Troubleshot production Airflow, Spark, BigQuery, and GCP pipeline failures, analyzing DAG execution logs, Spark stages, SQL performance, data anomalies, and dependency failures.

Senior Data Engineer

Kroger Co., Remote Dec 2022 to Dec 2023

Responsibilities:

Engineered large-scale GCP data pipelines using Java, Apache Spark, PySpark, Airflow, BigQuery, GCS, Dataproc, Pub/Sub, and Dataflow for retail, customer, product, transaction, inventory, and supply-chain datasets.

Developed Java/Spark distributed processing jobs to transform high-volume transactional data, implementing scalable joins, aggregations, filtering, deduplication, and business-rule transformations.

Built production Airflow DAGs for scheduling and orchestrating multi-step ETL/ELT pipelines across GCS, Dataproc, BigQuery, and downstream data consumers.

Implemented batch and streaming pipelines using Spark, Pub/Sub, and Dataflow to process transaction, inventory, product, and operational event data.

Developed advanced BigQuery SQL for data transformation, analytical processing, aggregation, reconciliation, and dimensional data modeling.

Created reusable ingestion frameworks for structured and semi-structured data using GCS landing zones, schema validation, metadata processing, and Spark-based transformations.

Tuned Spark workloads on Dataproc using partitioning, broadcast joins, caching, repartitioning, predicate pushdown, and optimized data layouts to improve processing performance.

Developed incremental ETL/ELT pipelines using watermarking, business keys, change detection, partition-based processing, and restartable Airflow workflows.

Implemented monitoring, logging, retry, alerting, and exception-handling mechanisms across Airflow DAGs, Spark jobs, and GCP data pipelines.

Performed production support and root-cause analysis for Spark, Airflow, BigQuery, Dataproc, and GCS failures, resolving pipeline latency, data-quality, dependency, and processing issues.

Data Engineer

Precision Healthcare Delivery, Sep 2021 to Dec 2022

Responsibilities:

Developed scalable GCP-based healthcare data pipelines using Java, Apache Spark, PySpark, Airflow, BigQuery, GCS, Dataproc, Pub/Sub, and Dataflow for patient, provider, claims, referral, scheduling, and care-delivery datasets.

Built distributed Spark processing frameworks to transform high-volume healthcare claims, eligibility, patient, provider, clinical encounter, and referral data.

Created Airflow DAGs to orchestrate ingestion, transformation, validation, aggregation, and publishing workflows across GCP data services.

Developed complex BigQuery SQL for healthcare analytics, including window functions, multi-table joins, claims aggregation, provider attribution, utilization analysis, and longitudinal patient transformations.

Developed Java-based Spark transformations for large-scale healthcare datasets, applying distributed processing patterns for scalable ETL workloads.

Designed GCS Dataproc/Spark BigQuery data flows for raw ingestion, standardized transformation, curated datasets, and downstream analytics.

Implemented data-quality and reconciliation frameworks covering patient identifiers, member eligibility, provider records, claims, diagnosis, procedures, referrals, and clinical encounters.

Optimized Spark pipelines using partitioning, predicate pushdown, broadcast joins, caching, repartitioning, and shuffle optimization to improve large-volume healthcare processing.

Implemented incremental processing and recovery mechanisms within Airflow and Spark, including retries, dependency controls, audit tracking, error handling, and restartable workloads.

Monitored and resolved production failures across Airflow, Dataproc, Spark, BigQuery, GCS, and Dataflow, performing root-cause analysis and implementing permanent pipeline fixes.

Data Engineer

Pharmacord LLC, Kentucky July 2020 to Sep 2021

Responsibilities:

Developed enterprise Azure ETL/data engineering pipelines using Azure Data Factory, Databricks, PySpark, Python, SQL, ADLS Gen2, and Delta Lake for healthcare payer and provider datasets.

Designed ADF pipelines for ingestion and orchestration of claims, eligibility, member, provider, reimbursement, diagnosis, procedure, utilization, and quality data.

Built Databricks processing workflows using PySpark and Spark SQL to transform high-volume healthcare datasets into curated analytical data layers.

Implemented Bronze/Silver/Gold architecture using Databricks and Delta Lake for scalable healthcare data ingestion, transformation, standardization, and analytics.

Developed Python/PySpark ETL logic for claims processing, eligibility matching, provider attribution, reimbursement calculations, utilization metrics, diagnosis/procedure transformations, and quality measures.

Performed detailed data validation and quality checks including source-to-target reconciliation, record-count comparison, duplicate identification, null checks, schema validation, referential integrity, and aggregate validation.

Developed automated validation routines to identify claims discrepancies, missing eligibility records, invalid provider relationships, inconsistent financial amounts, and transformation-level data issues.

Troubleshot ADF pipeline failures, Databricks job errors, Spark performance issues, schema mismatches, malformed records, and data-quality exceptions across development and production environments.

Optimized large-scale PySpark processing using partitioning, broadcast joins, caching, predicate pushdown, Spark SQL optimization, and Delta Lake optimization.

Participated in Agile development, collaborating with healthcare business teams, data engineers, QA, analysts, and technical stakeholders while independently managing ETL development, validation, troubleshooting, and production support.

Data Engineer

Imperium Health Management, Kentucky Sep 2019 – June 2020

Responsibilities:

Developed cloud-based GCP data pipelines using Java, Apache Spark, PySpark, Airflow, BigQuery, GCS, Dataproc, Pub/Sub, Dataflow, and SQL for specialty-pharmacy and pharmaceutical datasets.

Built scalable Spark ETL pipelines processing patient enrollment, prescription, dispensing, medication, shipment, provider, adherence, and specialty-pharmacy transaction data.

Developed Java/Spark distributed applications for large-volume data transformation, implementing complex joins, aggregations, filtering, deduplication, and business-rule processing.

Created Airflow DAGs to orchestrate pharmacy data ingestion, Spark transformations, BigQuery processing, data-quality checks, and downstream data delivery.

Developed complex BigQuery SQL for prescription fulfillment, dispensing activity, medication utilization, patient adherence, provider analytics, and pharmacy operations.

Designed GCS-based ingestion pipelines supporting structured and semi-structured pharmaceutical data, followed by distributed processing using Dataproc/Spark.

Implemented batch and event-driven workflows using Spark, Pub/Sub, and Dataflow to support scalable processing of pharmacy and operational events.

Optimized Spark jobs using partitioning, broadcast joins, caching, predicate pushdown, repartitioning, and shuffle optimization for high-volume prescription and patient datasets.

Implemented data-quality, reconciliation, and exception-handling frameworks across patient, prescription, dispensing, medication, provider, shipment, and adherence datasets.

Monitored and troubleshot Airflow DAGs, Spark jobs, Dataproc workloads, BigQuery SQL, and GCP data pipelines, resolving production failures and improving pipeline reliability and SLA performance.

Clinical Data Engineer March 2017 – August 2019

IUPUI, Indiana

Responsibilities:

Collaborated with clinical teams to gather and interpret study protocols, ensuring accurate translation into data requirements and specifications.

Designed, developed, and maintained clinical databases using SQL, SAS, and Excel to ensure high-quality, accurate, and compliant data capture.

Performed data validation, cleaning, and reconciliation for clinical trial datasets, ensuring adherence to regulatory and GCP standards.

Conducted exploratory data analysis, statistical reporting, and trend identification to support clinical decision-making and trial outcomes.

Developed dashboards and visualizations in Tableau, Power BI, or Python to summarize patient demographics, adverse events, and clinical outcomes.

Extracted and transformed data from multiple sources including EHR systems, lab reports, and clinical trial management systems (CTMS).

Applied statistical techniques such as regression analysis, ANOVA, survival analysis, and hypothesis testing to evaluate clinical outcomes.

Generated datasets and reports for regulatory submissions, audit readiness, and internal performance monitoring.

Assisted in creating SOPs, CRFs, and metadata dictionaries to standardize clinical data collection and reporting.

Supported cross-functional teams (medical, operations, biostatistics) with ad-hoc analyses to improve patient care and operational efficiency.

Participated in database migration, system upgrades, and integration projects, ensuring clinical data continuity and accuracy.

Clinical Data Analyst Apollo Hospitals, Indiana May 2013 – March 2017

Responsibilities:

Defined the data requirements, expected challenges and the required analytics techniques for the accomplishment of the project before initiating the analysis.

Collected & refined data (Data integrating) of Patient’s suffering with Cardiac Arrhythmia from organization’s Central Electronic health Records, cost records and caregiver’s satisfaction surveys.

Performed Data Auditing & Data Quality Assurance based on the Hospital’s standards using MS Excel.

Employed Python and MySQL language to subset data by removing outliers & missing values to create the final dataset consisting of 32% of elected primitive patient records with 47 attributes and 20 million records.

Calculated Descriptive statistics (Exploratory data analysis) to brief selected population by utilizing MySQL and Python, performed Kruskal-Wallis test on continuous variables and chi-square test on categorical variables.

Assisted in creating complex SQL formulas and creating excel spreadsheets.

Visualized Patient’s clinical outcomes and survival rates utilizing using Sankey diagrams, seaborn plots, Origin-Destination matrices, heat maps, stacked bar plots, timeline violin plots & several other visualizations using Bokeh Python library, other python libraries and Power BI.

Creating staging DB, move all the business scope data to the staging database and migrate to Hybris using SSIS

Developed interfaces to process the SAP Sales Orders, SAP Accounts, Shipping acknowledgements and ASN confirmation.

Worked on dimensional modeling and maintained dimensions like Product, Customer, and Region as a part of Snowflake schema and loaded the data in the member fact tables.

Involved in building and maintaining SSIS Packages to import and export the data from various data sources based on design data models.

Created packages in SSIS with error handling as well as created complex SSIS packages using various transformations and tasks like Sequence Containers, for loop and For Each Loop Container, Execute-SQL, Execute Package, Send Mail, File System, Conditional Split, Data Conversion, Derived Column, Lookup, Merge, Merge Join, Row Count, Union All, flat file source and destination, OLE DB source and destination, excel source and destination etc.

Used DAX (Data Analysis Expressions) functions for the creation of calculations and measures in the SSAS Tabular Models.

Contact this candidate