Senior Databricks Data Engineer & Analyst

Mohammed Yaseen

Data Engineer Data Analyst SQL Python Snowflake Power BI AWS Business Intelligence

*********@*****.*** +1-786-***-**** USA Open to Relocate

PROFESSIONAL SUMMARY

Senior Databricks Data Engineer with 7+ years of experience designing, developing, and implementing enterprise-scale cloud data platforms, modern data Lakehouse architectures, and high-performance ETL/ELT solutions across Financial Services, Insurance, and Healthcare domains. Extensive hands-on expertise in building scalable data engineering solutions using Databricks Lakehouse Platform, Delta Lake, Delta Live Tables (DLT), Unity Catalog, Databricks Workflows, PySpark, Spark SQL, Python, and advanced SQL.

Developed and optimized PostgreSQL databases, stored procedures, views, and complex SQL queries to improve data retrieval performance and support enterprise reporting. Participated in database migration projects, migrating legacy data from on-premises databases to cloud-based platforms while ensuring data integrity and minimal downtime. Supported data validation and reconciliation activities during migration, resolving data quality issues and ensuring successful production deployment.

Strong experience architecting end-to-end data pipelines supporting batch and real-time processing using Azure Data Factory (ADF), Azure Event Hubs, Apache Kafka, Azure Data Lake Storage Gen2 (ADLS Gen2), Snowflake, and cloud-native services across Azure and AWS. Proven expertise in implementing Medallion Architecture (Bronze, Silver, Gold), Change Data Capture (CDC), incremental data ingestion, schema evolution, data governance, and enterprise data quality frameworks.

Highly experienced in optimizing large-scale Spark workloads, Delta Lake performance tuning, partitioning, Z-Ordering, caching, and distributed processing to deliver reliable, high-performing analytical solutions. Strong knowledge of dimensional data modeling, Star Schema, Snowflake Schema, enterprise metadata management, and scalable data architecture supporting business intelligence, advanced analytics, and AI-driven initiatives.

Demonstrated ability to partner with business stakeholders, architects, product owners, and engineering teams to translate complex business requirements into scalable cloud-native data solutions. Experienced leading architecture discussions, driving modernization initiatives, and implementing secure, governed enterprise data platforms while following Agile methodologies and DevOps best practices.

WORK EXPERIENCE

United Services Automobile Association (USAA), Data Engineer 01/2025 – Present Plano, TX

• Designed, developed, and implemented enterprise-scale Databricks Lakehouse solutions for financial and insurance data platforms, enabling scalable, high-performance analytics across multiple business domains.

• Architected modern Medallion Architecture (Bronze, Silver, Gold) using Delta Lake, improving data quality, governance, and downstream reporting capabilities.

• Built scalable ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Python to process multi-terabyte structured, semi-structured, and streaming datasets.

• Developed Delta Live Tables (DLT) pipelines for automated data ingestion, transformation, validation, and continuous data quality monitoring.

• Implemented Unity Catalog for centralized metadata management, fine-grained access control, data lineage, governance, and secure enterprise data sharing.

• Built and managed Databricks Workflows and Databricks Jobs to orchestrate end-to-end enterprise ETL pipelines with automated scheduling, dependency management, retries, and alerting.

• Developed reusable Databricks Notebooks, shared libraries, and modular frameworks to standardize enterprise data engineering practices across multiple projects.

• Developed Shell Scripts (Bash) to automate ETL execution, log monitoring, file transfers, and routine administrative tasks, reducing manual operational effort.

• Automated application deployment, environment configuration, and batch processing using Shell scripting, improving operational efficiency and reducing execution errors.

• Integrated Azure Data Factory (ADF) with Databricks notebooks to orchestrate enterprise-scale batch and incremental data pipelines.

• Designed high-performance ingestion pipelines using Azure Data Lake Storage Gen2 (ADLS Gen2) and Databricks Auto Loader, enabling efficient processing of continuously arriving datasets.

• Developed real-time streaming data pipelines using Azure Event Hubs, Apache Kafka, Spark Structured Streaming, and Delta Lake to process high-volume transactional and operational data.

• Built Change Data Capture (CDC) pipelines using Delta Lake MERGE, enabling efficient incremental data synchronization across enterprise data platforms.

• Implemented Slowly Changing Dimension (SCD Type I & II) strategies to maintain historical accuracy within enterprise dimensional models.

• Designed scalable dimensional data models including Star Schema and Snowflake Schema supporting enterprise reporting, regulatory compliance, and advanced analytics.

• Optimized Delta Lake performance using OPTIMIZE, Z-ORDER, partitioning strategies, caching, Adaptive Query Execution (AQE), and file compaction, significantly improving query performance and reducing execution times.

• Developed enterprise data quality frameworks implementing schema validation, null handling, duplicate detection, reconciliation, and automated exception reporting.

• Leveraged Spark SQL, PySpark DataFrames, and SQL optimization techniques to improve complex query performance and reduce processing latency.

• Built secure cloud-native data platforms integrating Databricks with Snowflake, Azure SQL Database, SQL Server, REST APIs, and third-party enterprise systems.

• Implemented enterprise security standards using Unity Catalog, Azure Active Directory, Azure Key Vault, RBAC, encryption, and secure credential management.

• Automated infrastructure provisioning and deployment using Azure DevOps, Git, CI/CD pipelines, Docker, Kubernetes, and Terraform, reducing manual deployment effort and improving release consistency.

• Collaborated closely with business stakeholders, solution architects, product owners, and data governance teams to translate complex business requirements into scalable enterprise data architectures.

• Led architecture discussions and modernization initiatives, replacing legacy ETL workflows with cloud-native Databricks Lakehouse solutions that improved scalability, maintainability, and operational efficiency.

• Worked extensively with insurance policy, claims, customer, underwriting, financial, and risk datasets to build trusted enterprise data assets supporting regulatory reporting and executive dashboards.

• Monitored production data pipelines using Databricks monitoring, Azure Monitor, Log Analytics, and automated alerting mechanisms, ensuring high availability and operational reliability.

• Performed root cause analysis, production support, performance tuning, and proactive optimization of enterprise ETL workloads, consistently meeting SLA requirements.

• Partnered with cross-functional Data Engineering, Analytics, DevOps, Security, and Business Intelligence teams within Agile/Scrum environments, participating in sprint planning, architecture reviews, code reviews, and release management.

Environment: Databricks Lakehouse, Delta Lake, Delta Live Tables (DLT), Unity Catalog, Databricks Workflows, Databricks SQL, PySpark, Spark SQL, Python, SQL, Azure Data Factory, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Event Hubs, Apache Kafka, Spark Structured Streaming, Snowflake, SQL Server, Azure SQL Database, AWS S3, Azure DevOps, Git, Docker, Kubernetes, Terraform, Azure Key Vault, Azure Active Directory, Power BI, Tableau, Agile Scrum.

Infinite Computer Solutions, Data Analyst 04/2024 – 12/2024 Rockville, Maryland

• Designed and implemented enterprise-scale Databricks Lakehouse solutions supporting cloud-native analytics, operational reporting, and enterprise data modernization initiatives.

• Developed high-performance ETL/ELT pipelines using Databricks, PySpark, Spark SQL, Python, and SQL to process structured, semi-structured, and streaming datasets from multiple enterprise systems.

• Built scalable Delta Live Tables (DLT) pipelines to automate ingestion, transformation, validation, and continuous monitoring of enterprise data assets.

• Designed and maintained Apache Airflow DAGs to orchestrate ETL workflows, schedule batch jobs, monitor pipeline execution, and automate dependency management.

• Implemented Airflow monitoring, alerting, and retry mechanisms to improve workflow reliability and ensure SLA compliance.

• Implemented Unity Catalog for centralized metadata management, fine-grained access control, enterprise data governance, lineage tracking, and secure collaboration across engineering teams.

• Created and managed Databricks Workflows and Jobs to orchestrate end-to-end production ETL pipelines with automated scheduling, dependency management, monitoring, and failure recovery.

• Designed Medallion Architecture (Bronze, Silver, Gold) for enterprise data processing, significantly improving data quality, consistency, and downstream analytical performance.

• Integrated Azure Data Factory (ADF) with Databricks notebooks to automate enterprise data ingestion, orchestration, and workflow execution.

• Built real-time streaming pipelines using Azure Event Hubs, Apache Kafka, Spark Structured Streaming, and Delta Lake to process high-volume operational and transactional data.

• Leveraged Databricks Auto Loader to efficiently ingest incremental data from Azure Data Lake Storage Gen2 (ADLS Gen2) with automatic schema evolution.

• Implemented Change Data Capture (CDC) pipelines using Delta Lake MERGE operations to synchronize source systems with enterprise data warehouses.

• Developed dimensional data models using Star Schema and Snowflake Schema supporting executive dashboards, regulatory reporting, and advanced business intelligence.

• Optimized Spark workloads through partitioning, caching, broadcast joins, Adaptive Query Execution (AQE), Z-Ordering, and Delta table optimization, reducing processing time and improving query performance.

• Designed reusable PySpark frameworks, utility libraries, and Databricks notebooks to standardize enterprise development practices and improve code reusability.

• Integrated enterprise datasets from SQL Server, Snowflake, REST APIs, Azure Blob Storage, and cloud-native applications into unified Databricks Lakehouse solutions.

• Developed enterprise data quality frameworks implementing validation rules, reconciliation, duplicate detection, exception handling, and automated audit reporting.

• Implemented secure access controls using Azure Active Directory, RBAC, Azure Key Vault, and Unity Catalog to ensure compliance with enterprise security standards.

• Automated deployment of Databricks notebooks, workflows, and infrastructure using Azure DevOps, Git, Terraform, Docker, and CI/CD pipelines.

Environment: Databricks Lakehouse, Delta Lake, Delta Live Tables (DLT), Unity Catalog, Databricks Workflows, Databricks SQL, PySpark, Spark SQL, Python, SQL, Azure Data Factory, Azure Event Hubs, Azure Data Lake Storage Gen2 (ADLS Gen2), Apache Kafka, Spark Structured Streaming, Snowflake, SQL Server, Azure SQL Database, Azure DevOps, Docker, Kubernetes, Terraform, Git, Azure Key Vault, Power BI, Tableau, Agile Scrum.

Accenture services Pvt Ltd, Data Analyst 07/2018 – 08/2022 USA

• Designed, developed, and supported enterprise data engineering solutions for large-scale Banking, Financial Services, Insurance, and Healthcare clients using modern cloud-based data platforms.

• Built scalable ETL/ELT pipelines using Databricks, PySpark, Spark SQL, Python, and SQL for processing high-volume structured and semi-structured enterprise data.

• Developed enterprise Delta Lake architectures implementing ACID-compliant storage, schema evolution, time travel, and high-performance data processing.

• Designed and implemented Delta Live Tables (DLT) pipelines to automate ingestion, transformation, validation, and monitoring of production datasets.

• Built and managed Databricks Workflows for orchestrating enterprise batch and streaming data pipelines with dependency management and automated recovery.

• Implemented Unity Catalog for enterprise-wide metadata management, governance, access control, and secure data sharing across multiple business units.

• Developed scalable data ingestion frameworks using Azure Data Factory, Databricks Auto Loader, Azure Blob Storage, and ADLS Gen2 to process enterprise data from on-premises and cloud sources.

• Built real-time streaming data pipelines using Azure Event Hubs, Apache Kafka, Spark Structured Streaming, and Delta Lake to support near real-time business reporting.

• Designed enterprise Medallion Architecture (Bronze, Silver, Gold) supporting standardized data engineering practices across multiple business domains.

• Developed robust CDC and incremental loading frameworks using Delta MERGE, enabling efficient synchronization between transactional systems and analytical platforms.

• Designed dimensional data models using Star Schema and Snowflake Schema to support enterprise reporting, self-service analytics, and regulatory compliance.

• Performed advanced Spark optimization through partitioning, broadcast joins, caching, Delta optimization, Adaptive Query Execution (AQE), and file compaction to improve application performance.

• Integrated Databricks with Snowflake, SQL Server, Oracle, REST APIs, cloud storage services, and enterprise applications to build unified analytical platforms.

• Developed enterprise data quality frameworks implementing reconciliation, profiling, validation, exception handling, and automated auditing processes.

• Automated infrastructure deployment and application delivery using Git, Azure DevOps, Jenkins, Docker, Kubernetes, Terraform, and CI/CD pipelines.

• Collaborated with business stakeholders, solution architects, and product owners to gather business requirements, evaluate technical alternatives, and design scalable enterprise data architectures.

• Led modernization initiatives migrating legacy ETL workloads to cloud-native Databricks Lakehouse architecture, reducing operational overhead and improving scalability.

• Participated in architecture reviews, code reviews, sprint planning, release management, and Agile ceremonies while ensuring adherence to enterprise development standards.

• Provided production support, incident resolution, root cause analysis, performance tuning, and ongoing optimization of enterprise data platforms.

• Mentored junior engineers on Databricks best practices, Spark optimization, ETL design patterns, cloud-native development, and enterprise data governance.

• Produced comprehensive technical documentation including solution architecture, deployment guides, operational runbooks, data lineage, governance documentation, and knowledge transfer materials.

Environment: Databricks Lakehouse, Delta Lake, Delta Live Tables (DLT), Unity Catalog, Databricks Workflows, Databricks SQL, PySpark, Spark SQL, Python, SQL, Azure Data Factory, Azure Event Hubs, Azure Data Lake Storage Gen2 (ADLS Gen2), Apache Kafka, Spark Structured Streaming, Snowflake, SQL Server, Oracle, AWS S3, Azure DevOps, Jenkins, Git, Docker, Kubernetes, Terraform, Power BI, Tableau, Agile Scrum.

EDUCATION DETAILS

Master of Science, Information Technology Management, Indiana Wesleyan University 04/2024 Marion, U

Contact this candidate