Data Engineer Processing

.

.

VENKATA SR. DATA MANGIPUDI ENGINEER

Leander, TX 78641 302-***-**** **************@*****.***

www.linkedin.com/in/venkatapmanigipudi

PROFESSIONAL

SUMMARY

• Cloud-focused Data Engineer with over 8 years of experience in enterprise IT, specializing in AWS-based data pipeline development and scalable architecture using Databricks Lakehouse and modern ETL tools.

• Proven expertise in designing and implementing end-to-end data workflows using AWS Glue, S3, Lambda, Step Functions, and Delta Lake within Databricks, ensuring secure, high-performance data processing.

• Strong hands-on experience in Delta Live Tables (DLT), Unity Catalog for access control, and the Medallion architecture (Bronze/Silver/Gold layers) to deliver governed, analytics-ready data products.

• Developed and orchestrated ETL pipelines across multi-cloud platforms using Databricks

(PySpark) and AWS services, integrating data from diverse sources and optimizing performance.

• Implemented Delta Sharing for seamless and secure data exchange across platforms, supporting real-time collaboration and data democratization.

• Proficient in Python, SQL, T-SQL, and PL/SQL, with advanced Spark programming and workflow optimization in AWS and Databricks environments.

• Built and maintained robust data lakes and warehouses using AWS S3, Azure Data Lake Gen2, and Azure Synapse, with extensive experience in data modeling

(Star/Snowflake), profiling, cleansing, and transformation.

• Experience managing large-scale, serverless data pipelines using AWS Glue and Lambda, enabling scalable and cost-efficient data processing workflows.

• Skilled in real-time and batch processing, including Spark Streaming and cloud-native orchestration tools.

• Ensured high data integrity and compliance by implementing automated validation checks, anomaly detection, and structured governance using Unity Catalog and CI/CD best practices.

• Delivered business intelligence solutions with Power BI and SSRS, enabling executive visibility into KPIs and operational metrics.

• Familiar with Agile development practices, cross-functional collaboration, and version control using Git, Azure DevOps, and TFS.

Strong background in healthcare data analytics, working with claims, membership, and utilization datasets to uncover trends and support data-driven decision-making. SKILLS Data Warehousing: Amazon Redshift,

Google BigQuery, Microsoft Azure, Amazon

S3, Teradata, IBM Db2

Data Processing ETL/ELT: Apache Airflow,

Informatica, Talend, Workato, Microsoft

SSIS, EMR, MuleSoft, Apache Spark, Kafka

.

.

SQL and Database Systems: Amazon RDS,

MySQL, Oracle Database, PostgreSQL,

Microsoft SQL Server

Data Modeling: ERwin Data Modeler, Oracle

SQL Developer Data Modeler, Lucidchart,

ER/Studio

Data Visualization: Tableau, Microsoft

Power BI, MicrosStrategy, SAP Lumira,

Snowsight

Data Security & Compliance: PHI, PII,

HIPAA, Tokenization

Data Quality and Governance: Erwin Data

Intelligence, SAP Master Data Governance

Version Control

Collaboration:Git, AWS CodeCommit, Azure

DevOps, Visual Studio Code, Confluence,

Jira, OneDrive

Project Management: Gantt Charts, Scrum,

Kanban, Microsoft Project, Jira, Asana

WORK HISTORY SR. DATA ENGINEER 02/2023 to Current

Apple, Austin, TX

• Designed and implemented scalable ETL pipelines using Databricks and AWS Glue, enabling seamless data ingestion, transformation, and integration across cloud platforms.

• Leveraged Delta Lake on Databricks to build robust Medallion architecture

(Bronze/Silver/Gold layers), ensuring ACID-compliant and real-time analytics-ready datasets.

• Applied Delta Sharing for secure, real-time data collaboration across teams and partners.

• Developed and orchestrated serverless data workflows using AWS Step Functions and Lambda, automating event-driven ETL jobs.

• Managed data storage using Amazon S3, ensuring scalable, cost-efficient data lake infrastructure.

• Utilized Unity Catalog in Databricks to enforce fine-grained access control, audit trails, and governance across multi-cloud environments.

• Optimized Spark workloads and resource utilization within Databricks clusters, improving processing time and cost efficiency.

• Created robust data quality checks, anomaly detection, and validation layers to maintain data integrity across pipeline stages.

• Integrated Databricks pipelines with Power BI to deliver real-time dashboards, KPIs, and business metrics that informed supply chain planning.

• Built and deployed reusable Python-based data transformation scripts for large-scale ingestion and processing of structured and semi-structured data.

• Maintained CI/CD workflows using Azure DevOps and Git, automating deployment of data jobs and infrastructure components.

• Supported multi-platform relational database systems including PostgreSQL, MySQL, SQL Server, and Oracle for hybrid data source integration.

• Worked in Agile sprints, collaborating closely with data scientists, architects, and analysts to deliver production-ready solutions aligned with business needs.

.

.

SR. DATA ENGINEER 08/2021 to 01/2023

Kohl's, Sacramento, CA

• Migrated legacy data warehouse workloads to a modern Lakehouse architecture on Databricks.

• Built automated ETL pipelines leveraging AWS Glue and Python, improving pipeline reliability and reducing manual intervention by 60%.

• Collaborated with data scientists to build analytical datasets and feature stores, enabling ML model deployment.

• Led data quality checks and governance implementation using Python and SQL- based frameworks.

DATA ENGINEER 09/2019 to 07/2021

Payless ShoeSource, Sacramento, CA

• Designed, defined, and planned a database according to the documentation and business needs.

• Developed various views and stored procedures to update the data using another database.

• Created heavy-duty TSQL to join data from various views and functions, including left join, inner join, and outer join.

• Worked on different types of transformations that are available in Power BI query editor.

• Created packages using SSIS for data extraction from Flat Files, Excel Files, OLEDB to SQL Server.

• Wrote Python routines to log into websites and fetch data for selected options.

• Collaborated with business stake to integrate various data sources into Business Objects, ensuring accurate and timely reporting by leveraging ETL processes.

• Created Datasets in T-SQL, and stored procedures for Reporting services. SR. SQL DEVELOPER 04/2017 to 08/2019

Alluma, Sacramento, CA

• Developed an Operational Dashboard on SSRS to provide performance Key Performance Indicators (KPIs) of different govt agencies within the Arizona Govt. for weekly review meetings and helped improve operational efficiency by 15%.

• Developed various T-SQL stored procedures, triggers, views, and adding/changing tables for data load, transformation, and extraction.

• Developed Python ETL services for data loading, file parsing, and capturing audit data.

• Created Packages by testing and Cleaning the Standardized Data by using tools in Data Flow Transformations (Data Conversion, Export Column, Merge join, Sort, Union All, Conditional Split, and more) for existing/new packages.

• Writing different validation Scripts to check the accuracy of data.

• Created and scheduled SSIS packages for running AM and PM feeds from various departments and multiple servers and resources to Development Servers. Logged various packages as well as individual tasks using SQL server log providers like text files, SQL server database, trace files, XML file.

.

.

SQL DEVELOPER 10/2015 to 02/2017

WIPRO, Mysore, India

• Analyzed the current business processes and recommended/developed ETL solutions to meet the client's needs.

• Developed various T-SQL stored procedures, triggers, views, and adding/changing tables for data load, transformation, and extraction.

• Validated Data Integrity between SQL Database and Oracle PL/SQL Database by conducting Unit testing, Integration testing, User Acceptance testing.

• Involved in optimizing code and improving efficiency in databases including re-indexing, updating statistics, recompiling stored procedures, and performing other maintenance tasks. Involved in performance tuning of the slow-running queries and stored procedures.

EDUCATION Bachelor of Science: Electrical Engineering, 06/2011 to 05/2015 GITAM University - Visakhapatnam, INDIA

.

Contact this candidate