Senior Data Engineer - Data Warehousing & ETL Expert
Resume:
SENIOR DATA ENGINEER
Accomplished data engineer with career success analyzing, designing, developing, testing and maintaining data infrastructure solutions. Foster customer relationships, lead teams, and collaborate across functions to identify requirements, manage, and execute projects, producing data insights and delivering models, reports, and dashboards that guide decision making, maximize performance, improve processes, enhance user experience, and achieve quality objectives to drive business growth. Core competencies include: data modeling ETL processes Data warehousing pipeline development software development life cycle (SDLC) requirements analysis data mapping testing customer support collaboration data ingestion dashboard development data transformation storage performance tuning DAX calculations presentation dimensional modeling data engineering SQL queries, views, triggers, stored procedures pre-hook / post-hook scripts ELB generative AI Large Language Model (LLM) langchain TECHNOLOGY
Operating Systems: Windows Sun Solaris Linux (Ubuntu), Languages: PL/SQL Python (Panda) Java Unix Schell Scripting Big Data Technologies: Spark Kafka Hive Azure Event Hub Version Control: Gitlab GitHub Azure DevOps
Cloud Platforms: AWS Azure Cloud EC2 Cloudfront RDS Databases: Oracle MS SQL Teradata DBT PostgreSQL Dynamo DB Redshift Hadoop Snowflake MongoDB NoSQL Lambda
ETL Tools: Informatica PowerCenter AWS Glue Databricks Azure Data Factory S3 SQS SSIS Data Visualization: Power BI Tableau
Data Modeling Tools: ERWIN Data Vault
EXPERIENCE
DynamoDB for low-latency data storage and retrieval.
• Designed and implemented data warehousing solutions using Snowflake including schema design and dimensional modeling.
• Created DBT jobs to automate deployment process.
• Conducted performance tuning and optimization of DBT models and SQL queries to improve query performance and reduce processing time. Created DBT Macros for reuse them in various models.
• Optimized Amazon EMR clusters for big data processing using Apache Spark and PySpark, reducing job execution time by 40%.
• Used Databricks to ingest datasets and Data frame API to read and write data into and from Azure data lake.
• Designed and implemented ETL/ELT processes using DBT to extract, transform, and load data from various source systems into the data warehouse.
• Built scalable ETL pipelines using AWS Glue to ingest and transform data from S3, RDS, and APIs into Amazon Redshift for analytics and reporting.
• Designed and implemented data pipelines using Data Build Tool (DBT) to transform raw data into analytics-ready datasets within Snowflake data warehouse.
• Used Apache Airflow to orchestrate the ETL pipeline. Shumet Nigatu
******.******@*****.*** www.linkedin.com/in/shumet Geico Insurance March 2020 - Present
Senior Data Engineer Chevy Chase, MD
• Extracted data from flat and relational data sources like Oracle, DB2 and SQL to build operational data sources.
• Developed serverless applications with AWS Lambda to automate data processing workflows, integrating with
• Experienced in Ingesting data from S3 to Azure blob storage(staging) and load the transformed dataset to ADLS in Azure Synapse.
• Partnered with business, architecture, and infrastructure team to develop service level agreements.
• Developed and optimized ETL workflows using PySpark and SQL for large scale data processing and optimization.
• Designed and optimized Snowflake data warehouses, including virtual warehouses, clusters, and storage configurations, to achieve optimal performance and cost efficiency.
• Developed and maintained a centralized data model for consistency across reports and dashboards.
• Implemented ELT processes within Snowflake using SQL-based transformations and Snowflake's native functions to cleanse, transform, and enrich data as it is loaded into data warehouses.
• Analyzed and classified complex change requests, identifying and documenting possible system code enhancements.
• Developed and maintained DBT models to perform complex data transformations, including data cleansing, enrichment, and aggregation, ensuring data accuracy and consistency.
• Developed Power BI reports and dashboards to monitor key performance indicators (KPIs) and trends.
• Ingested data from various sources to snowflake data warehouse using HVR and transform it using DBT.
• Developed and maintained Azure Databricks notebooks using Python (PySpark ) to ingest, transform, and load large- scale datasets into Delta Lake for efficient querying and analytics.
• Scheduled and automated jobs in Databricks using notebooks and Azure Data Factory.
• Created dashboards and alerts to visualize and monitor the performance of service in azure subscriptions.
• Used Kafka to stream live data from API to the Datawarehouse.
• Optimized Databricks clusters and Spark SQL jobs, improving query performance and reducing compute costs.
• Designed and deployed dataflows in Microsoft Fabric to streamline data preparation and transformation processes.
• Leveraged Microsoft Fabric’s OneLake to create a centralized, scalable, and secure data lake for enterprise-wide Teradata to flat file targets.
• Built serverless ETL pipelines using AWS Glue and Lambda, ingesting 10TB+ of policy data monthly from S3, RDS, and APIs into Redshift, reducing load times by 35%.
• Built event-driven architectures using S3 triggers, SQS, and Lambda for seamless data ingestion.
• Performed standard ETL administration such as deployment, package migration and relocation, and production support per business requirements.
• Implemented performance tuning of sources, targets, mapping, and sessions by identifying bottlenecks.
• Optimized Spark jobs and Databricks clusters to enhance workflow performance.
• Developed Shell scripts for various tasks like data quality and file transfer.
• Managed version control and deployment processes using Azure DevOps, ensuring high-quality, continuous integration and deployment.
• Orchestrated ETL pipelines with Prefect for scalable, fault-tolerant workflow automation.
• Maintained technical documentation and supported stakeholders to understand data structures and processes.
• Implemented transformation techniques in Databricks to ingest data from http endpoints and blob storage to ADLS.
• Used Azure Data Factory to extract, process and transform datasets from various sources, created pipelines and loaded the data into Azure delta lake.
EDUCATION
analytics.
• Built data pipelines in Microsoft Fabric to ingest, transform, and load data, enabling real-time and batch processing. Bank of America September 2017 – March 2020
Data Engineer Charlotee, NC
• Developed automated data validation scripts in Python and reduced data quality issues.
• Developed scalable ETL/ELT pipelines using Azure Data Factory, PySpark, and Databricks.
• Involved in writing queries, function, triggers, create views and stored procedures for data extraction and reporting.
• Designed, analyzed, implemented, tested, and supported ETL processes for stage, ODS and Mart.
• Worked on informatica code mapping requirements and loaded data from relational databases such as Master of Science in Computer Science University of Trento
Contact this candidate