Data Engineer Lead

Shauk Hassan

Lead Data Engineer Cloud & Data Platform Specialist Multi-Cloud Streaming & Lakehouse Expert

# **************@*****.*** 213-***-**** + West Chester, Ohio, 45071

§ https://github.com/Shauk-Hassan

https://shauk-hassan.github.io/portfolio/

Profile

Lead Data Engineer with 11+ years of experience designing, building, and scaling multi-cloud data ecosystems across AWS, Azure, and GCP. Specialized in high-performance ETL/ELT and real-time streaming architectures leveraging Kafka, Spark, and Snowflake to process over 5 million events per hour and 5 terabytes of data daily across healthcare, finance, and insurance domains. Skilled in Databricks, Delta Lake, and modern data modeling frameworks, driving governed and scalable data platforms that enable real-time decision intelligence. Led global data initiatives modernizing multi-cloud platforms, enforcing compliance, and improving reliability by 99%. establishing data governance and observability frameworks, and implementing CI/CD and IaC automation using Airflow, dbt, and Terraform to achieve 99% deployment reliability. Adept at leading cross-functional teams to deliver secure, compliant (HIPAA, GDPR, CCPA) solutions that cut infrastructure costs by 22% and reduce data incidents by 60%. Passionate about integrating GenAI and MLOps pipelines to unlock predictive insights and accelerate enterprise digital transformation.

Skills

Programming & Scripting:

Python, SQL, Scala, Java, C#, Shell scripting,.NET Data Engineering & Integration:

ETL/ELT, dbt, Airflow, Azure Data Factory, Informatica, Apache NiFi, Talend, Matillion, Fivetran, Prefect, Data Migration, Real- Time Data Processing, Pipeline Optimization

Data Warehousing & Modeling:

Snowflake, Redshift, BigQuery, Synapse, Teradata, Oracle, SQL Server, NoSQL, Dimensional & Relational Modeling, Star Schema, Data Vault, Data Mesh, Data Fabric,

Apache Iceberg, ORC, Reverse ETL

Cloud Platforms:

AWS (S3, Redshift, Glue, Lambda, EMR, Kinesis, Lake Formation, CloudFormation), Azure (ADF, Synapse, ADLS, Azure SQL, Functions), GCP (BigQuery, Dataflow,

Dataproc, Pub/Sub, Vertex AI), Databricks

MLOps & Advanced Analytics:

MLflow, TensorFlow, PyTorch, Scikit-learn, SageMaker Pipelines,Vertex AI, LLMOps, Databricks ML

Governance & Security:

HIPAA, GDPR, CCPA, IAM, RBAC, Data Masking,

Encryption, Data Catalogs (AWS Glue, Alation, Collibra), Master Data Management (MDM), Data Observability,

Data Lineage, Data Quality (Great Expectations, Monte Carlo) Big Data & Streaming:

Hadoop, Spark (Batch & Streaming), Kafka, Flink, Beam, Pulsar, HDFS, Hive, HBase, Presto, Delta Lake, Storm

DevOps & Automation:

Docker, Kubernetes, Jenkins, GitHub Actions, Azure DevOps, Ansible, Terraform, GitOps, Agile/Scrum, Automated Testing & Monitoring

BI & Reporting:

Tableau, Power BI, Looker, Superset, Mode Analytics, Custom & Interactive Dashboards

Domain Expertise:

Healthcare (EHR, Claims Data, HL7, Clinical Data

Integration, Quality Measures), Finance (Budgeting, Forecasting, Financial Modeling, Risk Management, Variance Analysis, KPI Analysis, ROI Calculation), Insurance, Manufacturing, Media/Entertainment

Professional Experience

MediTech 03/2021 – Present

Lead Data Engineer

• Architect scalable multi-cloud data platforms on AWS, Azure, and GCP, integrating healthcare and finance workloads under a unified lakehouse ecosystem.

• Build real-time streaming pipelines with Kafka and Spark, processing 5M+ events per hour with sub-3s latency for mission-critical analytics.

• Design and optimize Snowflake warehouses with CDC pipelines (Snowpipe, Streams, Tasks) for real-time ingestion and partner data exchange, cutting query costs by $1.2M annually.

• Modernize batch and streaming pipelines with Delta Lake and Apache Iceberg, improving query performance 3 and reducing data duplication.

• Establish observability, data quality, and lineage frameworks, increasing SLA adherence 35% and ensuring HIPAA, GDPR, and CCPA compliance.

• Deploy AI and ML pipelines for fraud detection and patient-risk prediction, enhancing model accuracy 30% and automating decision workflows.

• Automate CI/CD and IaC pipelines with Airflow, dbt, and Terraform, reducing deployment time 60% and achieving 99% release reliability.

• Integrate Power BI and Power Platform (Power Apps, Power Automate) with Synapse pipelines, eliminating 50% of manual reporting.

• Standardize reusable data models and ETL frameworks across domains, accelerating project onboarding 40% and improving consistency.

• Integrate Databricks MLflow with Synapse and Snowflake for continuous retraining and production-scale AI deployment.

• Lead and mentor a 10-member engineering team, improving CI/CD success from 72% to 99% and increasing throughput 45%.

• Optimize multi-cloud infrastructure, reducing spend 22% while improving performance and scalability.

• Deliver self-service analytics platforms enabling 300+ users to access governed, real-time insights without engineering dependency.

Trigent Software 09/2017 – 02/2021

Senior Data Engineer

• Developed ETL/ELT pipelines with Airflow and dbt, orchestrating 1PB+ data across finance and payments.

• Created CDC pipelines with Kafka + Debezium, cutting latency from 24 hours to under 2 minutes.

• Migrated pipelines to AWS Glue (PySpark Jobs Crawlers) and automated ingestion with AWS Lambda and Step Functions, while modernizing analytics workflows in BigQuery and Databricks, accelerating campaign insights by 65%.

• Built observability and validation frameworks using Great Expectations and Monte Carlo, improving SLA adherence to 98% and reducing downstream errors by 45%.

• Integrated MLOps pipelines with Vertex AI, expediting recommendation deployments by 60%.

• Deployed Delta Lake on Databricks, tripling query performance for analysts and auditors.

• Delivered predictive pipelines that generated $5M+ savings in payments infrastructure.

• Established CI/CD with Jenkins + GitHub Actions, moving releases from monthly to weekly.

• Mentored junior engineers on Python, Spark, and SQL, elevating code quality and reducing review cycles by 25%.

• Built financial data pipelines integrating Netsuite and Workday with Snowflake and dbt, automating ERP reporting workflows and enabling reusable schema libraries across 50+ datasets for consistent portfolio analysis and governance.

• Constructed data quality monitoring dashboards that improved SLA adherence from 80% to 98%.

• Facilitated cross-functional workshops with data science teams, reducing ML model deployment times by 40%. SumatoSoft 05/2015 – 08/2017

Cloud Data Engineer

• Orchestrated ingestion pipelines on AWS Glue and Lambda, processing 20TB/day of insurance and IoT data.

• Delivered Power BI dashboards for risk analysis and predictive maintenance, lowering downtime costs by 22%.

• Migrated legacy warehouses to Azure Synapse, trimming infrastructure costs by 30%.

• Executed cross-cloud replication (AWS S3, Azure ADLS, GCP Storage), sustaining 99.9% uptime for disaster recovery.

• Devised monitoring with CloudWatch and Azure Monitor, cutting incident response time by 40%.

• Automated governance dashboards with Terraform and CloudFormation, reducing infra spend by 20%.

• Formulated high-availability data lake with Delta Lake, boosting scalability and minimizing query failures by 35%.

• Directed multi-cloud security hardening initiatives and applied RBAC/ABAC policies in Azure Entra ID, ensuring SOC2, HIPAA, NIST 800-53, and FedRAMP compliance.

10Pearls 01/2014 – 04/2015

ETL & Data Warehouse Specialist

• Constructed ETL workflows with Informatica and SSIS, processing 500M+ records daily across retail and telecom data warehouses, while building Epic Clarity/Caboodle healthcare integrations and developing metadata catalogs and dictionaries in AWS Glue and Alation, reducing batch load times by 35%.

• Streamlined batch job scheduling, eliminating 70% of manual operations in telecom billing.

• Tuned POS reporting queries, shrinking runtime from 8 hours to 1 hour.

• Secured pipelines with HIPAA, GDPR, and CCPA compliance by implementing IAM, encryption, and data masking, and standardized healthcare pipelines with ICD-10, CPT, LOINC, and SNOMED coding frameworks.

• Transitioned BI workloads from SSRS to Tableau, increasing analyst adoption by 45%.

• Designed KPI dashboards for retail forecasting and churn prediction, driving executive insights. Projects

Unified Multicloud Data Mesh & AI Platform

• Architected cross-cloud framework unifying healthcare, retail, and finance data with full HIPAA/GDPR compliance.

• Built GenAI-assisted pipelines (Databricks, Snowflake, Kafka, Airflow, dbt), delivering 30% cost reduction and faster analytics adoption.

Financial Risk & Compliance Lakehouse

• Re-engineered financial risk platform on Azure Databricks/Synapse, cutting SLA violations by 40%.

• Operationalized ETL/ELT pipelines, enabling 70% faster portfolio analysis across global finance teams. Healthcare Cost Optimization & Modernization

• Migrated legacy warehouses to Teradata, Netezza, and Redshift, doubling throughput and performance.

• Integrated governance and HIPAA frameworks, driving annual savings of $20M+ in operational costs. Legacy-to-Modern Data Platform Migration

• Modernized 300+ legacy ETL pipelines from Teradata/Oracle to Snowflake & Databricks with zero data loss. 70% faster queries and $2M annual savings, maintaining enterprise compliance. Real-Time Customer Personalization Engine

• Prototyped streaming personalization layer (Kafka, Flink, Delta Lake, Vertex AI, MLflow), processing 2M+ events/hour.

• Increased relevance of recommendations by 28%, driving measurable engagement boost in e-Commerce and media pilots.

Certifications

• AWS Solutions Architect – Professional

• Google Cloud Professional Data Engineer

• Microsoft Azure Solutions Architect – Expert

• Databricks Data Engineer – Professional

• Snowflake SnowPro Advanced – Architect

• Cloudera CCP – Data Engineer

• Kubernetes Administrator (CKA)

• Apache Kafka Developer Certified

Education

Bachelor of Science (B.S.) in Computer Science

Contact this candidate