Senior Data Engineer - Databricks Lakehouse

Mustafa M

Senior Data Engineer Data Architecture, Big Data Python, Databricks

*******.*****@*****.*** +1-832-***-**** Richmond, TX 77407 Summary

Data Engineer with 8+ years of experience across SaaS, FinTech, Healthcare, and Technology domains, building scalable cloud-native data platforms and modern data ecosystems. Strong expertise in Databricks, Snowflake, and AWS for designing high-performance ETL/ELT pipelines, real-time streaming architectures, and Lakehouse solutions. Experienced in working with large-scale distributed data using Spark and Python, supporting analytics, BI, and AI/ML use cases. Proven track record in implementing data quality, governance, and DataOps practices while optimizing performance and cost across enterprise data platforms. Professional Experience

Senior Data solution Architect

ACE

02/2024 – Present

•Working in healthcare and education domain, building scalable data platforms processing 5+ TB daily data supporting student outcomes and analytics.

•Designed Lakehouse architecture using Databricks, Delta Lake, and AWS S3, improving data access across structured and unstructured datasets.

•Built real-time pipelines using Kafka and Spark Structured Streaming, processing millions of events/day with faster data availability.

•Developed ELT pipelines using PySpark, SQL, and dbt, improving performance by 25% across large-scale workflows.

•Enabled AI/ML workflows by building curated datasets supporting prediction and analytics models.

•Integrated APIs and external systems handling 100+ data feeds, reducing ingestion failures by 30%.

•Used AI tools like GitHub Copilot and Databricks Assistant to improve development speed and efficiency. Senior Data Engineer

GrowthLoop

08/2020 – 01/2024

•Worked in SaaS and marketing analytics domain, building customer data platforms supporting millions of user profiles, improving campaign targeting.

•Built scalable data pipelines using Snowflake, dbt, and Airflow, processing multi-terabyte datasets daily with high reliability.

•Developed event-driven ingestion frameworks using APIs and Kafka, enabling near real-time data processing for marketing events.

•Designed data models and transformation layers, improving query efficiency by 28% and enhancing dashboard performance.

•Enabled machine learning use cases by preparing structured datasets supporting segmentation and predictive models.

•Implemented data validation, monitoring, and lineage, reducing data quality issues by 32% and improving system reliability. Data Engineer

Nerdery

03/2018 – 07/2020

•Worked in consulting across healthcare and fintech domains, delivering scalable solutions.

•Built ETL/ELT pipelines using Python, Spark, and AWS services, handling high-volume batch data workloads and reducing processing time.

•Developed secure data pipelines for healthcare systems, ensuring HIPAA aligned compliance and data protection standards.

•Designed and implemented data warehouses supporting analytics, reporting, and regulatory requirements.

•Built API-driven ingestion frameworks integrating enterprise and third-party systems with reliable data flow.

•Optimized SQL queries and Spark jobs, improving performance reducing costs.

•Collaborated with cross-functional teams to deliver analytics solutions, increasing adoption of data-driven decision making. Education

Bachelor of Science

Bachelor of Science in Computer Science

University of Houston

Skills

Programming

Python, Advanced SQL, Bash scripting

Streaming & Messaging

Apache Kafka, AWS Kinesis, Event-driven Architecture, Real-time Data Ingestion, Stream Processing

Big Data & Processing

Apache Spark, PySpark, Spark SQL, Spark Structured Streaming, Distributed Processing, Batch & Real-time Pipelines, Performance Tuning, Partition Optimization

Orchestration & Automation

Apache Airflow (DAGs, Scheduling, SLA Monitoring), Terraform, CI/CD (GitHub Actions), Git, Jenkins

Monitoring & Optimization

CloudWatch, Logging Frameworks, SLA Monitoring, Cost Optimization, Performance Tuning, Resource Optimization Databases & Storage

Snowflake, Amazon Redshift, PostgreSQL, MySQL, NoSQL

(MongoDB), Data Lakes (S3, ADLS)

Cloud & Data Platforms

AWS (S3, Glue, EMR, Lambda, Redshift, Kinesis, IAM), Azure

(ADLS, Data Factory, Synapse), Databricks, Snowflake, Delta Lake, Lakehouse Architecture, Data Lake Design

ETL/ELT & Data Engineering

ETL/ELT Pipeline Design, Incremental Loading, CDC (Change Data Capture), Data Transformation, API Ingestion, Data Reconciliation

Data Warehousing & Modeling

Dimensional Modeling, Star/Snowflake Schema, SCD Type 1/2, Data Marts, Query Optimization, Semantic Layer Design Data Quality & Governance

Great Expectations, Data Validation Rules, Schema Enforcement, Data Lineage, PII Masking, HIPAA Compliance, Data Cataloging AI/ML & Modern Data Tools

MLflow, Feature Store, AI-assisted Development (GitHub Copilot, Databricks Assistant), DataOps, LLM-assisted Query Optimization

Certificates

AWS Certified Data Analytics – Specialty

Databricks Certified Data Engineer Associate

AWS Certified Solutions Architect – Associate

Azure Data Engineer Associate

Projects

Healthcare Outcomes & Compliance Analytics Platform

•Built a centralized data platform integrating clinical, student, and operational datasets to support outcome tracking, regulatory reporting, and compliance analytics.

•Tech Stack: Databricks, Delta Lake, PySpark, AWS S3, Airflow, Great Expectations, HIPAA-compliant architecture Data Contract & Schema Governance Framework

•Designed a framework to enforce data contracts and manage schema evolution across pipelines, improving data consistency and reducing downstream failures.

•Tech Stack: Python, dbt, Databricks, Delta Lake, JSON Schema, Airflow Synthetic Data Generation & Test Data Platform

•Built a platform to generate realistic synthetic datasets for testing and development, enabling safe use of healthcare and sensitive data without exposing PII.

•Tech Stack: Python, Spark, Databricks, Faker, Snowflake, AWS S3, Data Masking Techniques

Contact this candidate