Senior Cloud Data Engineer, Big Data

Naveena Reddy

Email: ***************@*****.*** Mobile: 209-***-****

LinkedIn: https://www.linkedin.com/in/naveena-reddie/

Professional Summary:

Senior Data Engineer with 9+ years of IT experience specializing in Big Data, data engineering, cloud data platforms, data analytics, and enterprise data solutions. Proven experience designing, developing, and deploying scalable, high-performance data pipelines and mission-critical applications using Python, PySpark, Scala, Apache Spark, SQL, Hadoop, Hive, Kafka, and distributed data processing technologies.

Extensive experience building modern cloud-based data platforms using Azure Databricks, Azure Data Factory, Microsoft Fabric, Azure Synapse Analytics, AWS, and Snowflake, with strong expertise in ETL/ELT, CDC, data ingestion, transformation, orchestration, data quality, data cleansing, and performance tuning.

Hands-on experience with Apache Airflow, Kafka, Spark Streaming, HDFS, Map Reduce, YARN, Hive, HBase, Cassandra, Sqoop, Oozie, and Zookeeper for batch and real-time data processing.

Strong background in data modelling, dimensional modelling, data warehousing, data lakes, lakehouse architecture, Medallion Architecture, Delta Lake, NoSQL, relational databases, and enterprise data integration.

Experienced in data governance, data lineage, cataloging, classification, metadata management, compliance, and data quality using Microsoft Purview, Collibra, and Erwin.

Experienced in Generative AI, Large Language Models (LLMs), Azure OpenAI, NLP, Machine Learning, RAG concepts, semantic search, and AI-ready data pipelines, including preparation of structured and unstructured data for AI/ML and GenAI applications.

Proficient in Python data science libraries including Pandas, NumPy, SciPy, Scikit-learn, Matplotlib, and NLTK.

Strong experience with AWS EMR, EC2, S3, Redshift, Azure cloud services, Informatica, Talend, IBM Data Stage, and Hadoop distributions including Cloudera, Hortonworks, MapR, and IBM Big Insights.

Skilled in CI/CD and DevOps using Jenkins, GitHub Actions, GitLab CI/CD, Azure DevOps, Docker, and Kubernetes, along with scripting and automation using Python, Shell, PowerShell, Ruby, Perl, YAML, and Groovy.

Experienced in delivering financial services, regulatory reporting, risk analytics, enterprise data, and business intelligence solutions, with hands-on expertise in Power BI, Tableau, and Pentaho.

Strong understanding of SDLC, Agile, and Waterfall methodologies, with a proven ability to collaborate with business stakeholders, data architects, solution architects, and cross-functional development teams to deliver reliable, scalable, and business-driven data solutions.

TECHNICAL SKILLS:

Programming Languages: Python, Scala, Java, R, PL/SQL, SQL, HiveQL, Pig Latin, C, C++, Shell Scripting

Big Data & Hadoop Ecosystem: HDFS, YARN, MapReduce, Hive, Pig, Kafka, Flume, Sqoop, Oozie, ZooKeeper, HBase, Cassandra, Apache NiFi, Impala, MongoDB

Apache Spark: Spark Core, Spark SQL, DataFrames, Spark Streaming, MLlib, PySpark, Scala

AI / ML / GenAI: Generative AI, Large Language Models (LLMs), Azure OpenAI, RAG, NLP, Machine Learning, PySpark MLlib, Scikit-learn

Databases: Oracle, SQL Server, MySQL, PostgreSQL, Teradata, HBase, Cassandra, MongoDB, NoSQL

Cloud Platforms: AWS (EMR, EC2, S3, Redshift, RDS, Lambda, Data Pipeline), Microsoft Azure

ETL & Data Warehousing: Informatica, Talend, IBM DataStage, Pentaho, SAP BusinessObjects (SAP BO), ETL/ELT, Data Warehousing, Data Modeling

Data Modeling & Architecture Tools: Erwin, ER/Studio, IBM InfoSphere Data Architect, PowerDesigner

Data Governance & Management: Microsoft Purview, Data Cataloging, Data Lineage, Metadata Management, Data Classification, Data Quality

DevOps & Build Tools: Git, SVN, Jenkins, Maven, Docker, Kubernetes, Azure DevOps, GitHub Actions, GitLab CI/CD

Workflow, Orchestration & Streaming: Apache Airflow, Kafka, Spark Streaming, Oozie, Apache NiFi, ZooKeeper

Data Visualization & BI: Power BI, Tableau, Pentaho

Operating Systems: Linux, UNIX, Windows

Development & Productivity Tools: Eclipse, NetBeans, Microsoft Office, Microsoft Visio, Microsoft Project

Professional Experience:

Client: M&T Bank, Bridgeport, CT

Jan 2025 – Present

Role: Senior Data Engineer

Responsibilities

Analysed business requirements, created technical specifications, source-to-target mappings, data models, logical/physical schemas, data dictionaries, and validation rules using ER/Studio, supporting scalable enterprise data solutions.

Designed and developed real-time and batch data pipelines using Apache Spark, Scala, PySpark, Kafka, HDFS, Hive, Python, Apache NiFi, and HBase for high-volume distributed data processing.

Designed and optimized scalable ETL/ELT pipelines using Pentaho PDI, Azure Data Factory, Apache Airflow, and cloud-native services, improving processing efficiency by approximately 40%.

Developed Azure Data Factory pipelines with dynamic parameterization, reusable components, monitoring, error handling, scheduling, and comprehensive data quality and validation for multi-terabyte datasets.

Designed Medallion Architecture (Bronze, Silver, Gold) using Azure Databricks and Delta Lake, supporting scalable data lakehouse, analytics, and reporting solutions with Spark optimization, cluster management, security, and performance tuning.

Built modern data pipelines using Microsoft Fabric and Azure Synapse, enabling cloud analytics, data integration, fault-tolerant processing, and enterprise data platform architectures.

Developed data pipelines supporting financial services, regulatory reporting, risk analytics, and compliance use cases, including large-scale financial and time-series data processing using kdb+ / kdb Insights, Service Gateway (SG), DAPs, partitioning, and query optimization.

Designed AI-ready data pipelines for Generative AI, LLMs, Azure OpenAI, NLP, and Machine Learning, including data cleansing, chunking, metadata enrichment, embedding generation, vector search, semantic retrieval, prompt/context preparation, response validation, and governance.

Developed Python-based data processing pipelines using Pandas, NumPy, Scikit-learn, TensorFlow, NLTK, and spaCy for JSON processing, predictive analytics, NLP, feature preparation, and intelligent data solutions.

Designed and maintained data governance frameworks using Collibra and Erwin, supporting data quality, metadata management, data lineage, classification, compliance, and governance.

Worked with MongoDB and Amazon DocumentDB, designing scalable NoSQL schemas, optimizing queries, and leading MongoDB-to-DocumentDB migration with data integrity, replication, high availability, and failover.

Developed Snowflake pipelines using Snowpipe and SQL, implementing data loading, transformation, optimization, and performance tuning; also utilized AWS S3, EC2, EMR, Redshift, Lambda, IAM, and CloudFormation for scalable cloud data solutions.

Implemented GCP BigQuery, Apache Beam, and Dataflow pipelines for data transformation, validation, reconciliation, source-to-target consistency, and scalable analytics.

Implemented Apache Airflow for workflow orchestration, dependency management, scheduling, monitoring, and automated execution of complex data pipelines.

Developed CI/CD and DevOps solutions using Jenkins, GitHub Actions, GitLab CI, Git, Docker, Kubernetes, and automated testing, supporting consistent deployments across development, QA, and production.

Developed and deployed containerized applications using Docker and Kubernetes, and automated data engineering workflows using Python, Shell scripting, and PowerShell.

Created Power BI and Pentaho BI dashboards and reports with KPI reporting, secure access controls, data governance, and operational analytics; also supported Tableau reporting solutions.

Performed API/UI test automation using REST Assured and Selenium, and developed enterprise applications using J2EE, Servlets, JSP, JDBC, deployed on Tomcat and JBoss.

Optimized Hive, Spark, SQL, and database queries, applying distributed processing and performance-tuning techniques to improve large-scale analytical workloads.

Environment: Python, Scala, SQL, PySpark, Apache Spark, Hadoop, HDFS, Hive, Kafka, HBase, NiFi, kdb+, kdb Insights, Azure Data Factory, Azure Databricks, Azure Synapse, Microsoft Fabric, Delta Lake, AWS S3, EC2, EMR, Redshift, Lambda, IAM, CloudFormation, GCP BigQuery, Apache Beam, Dataflow, Snowflake, Snowpipe, MongoDB, Amazon DocumentDB, Apache Airflow, Pentaho, Collibra, Erwin, ER/Studio, Power BI, Tableau, Docker, Kubernetes, Jenkins, GitHub Actions, GitLab CI, Git, Machine Learning, Generative AI, LLMs, Azure OpenAI, NLP, TensorFlow, NLTK, spaCy, Scikit-learn, Pandas, NumPy, REST Assured, Selenium, J2EE, Servlets, JSP, JDBC, Tomcat, JBoss, Shell, PowerShell

Cognizant, India

Nov 2022 – Jun 2023

Role: Senior Data Engineer

Responsibilities:

Designed and developed scalable ETL/ELT data pipelines using Microsoft Fabric, Azure Data Factory, Databricks, Apache Spark, PySpark, Python, Informatica, and Pentaho, supporting large-scale financial and enterprise datasets.

Reduced ETL processing time by approximately 45% through Spark optimization, partition pruning, workload parallelization, query tuning, and efficient data processing strategies.

Implemented Microsoft Fabric solutions using Lakehouse, Warehouse, Dataflows Gen2, Event Streams, Real-Time Hub, KQL Databases, and Spark Notebooks for batch and real-time analytics.

Designed Direct Lake, Import, and Hybrid Semantic Models using Power BI and DAX, implementing Star and Snowflake schemas for enterprise reporting and analytics.

Developed CDC-based ingestion pipelines supporting near real-time analytics with data latency below five minutes.

Optimized Apache Spark and Microsoft Fabric workloads through Spark configuration tuning, partitioning, monitoring, observability, and performance optimization.

Implemented CI/CD pipelines for Fabric workspaces using Azure DevOps, reducing deployment effort by approximately 60%, and automated notebook testing and validation.

Built Snowflake and Redshift pipelines using Snowpipe, Snowpark, SQL, and Python, optimizing data ingestion, transformation, and warehouse performance.

Developed real-time streaming pipelines using Kafka, Spark Streaming, Scala, and PySpark, processing data into HDFS/Hive environments.

Designed scalable data lakehouse and enterprise data models using Databricks, ER/Studio, dimensional modeling, metadata management, and layered architecture patterns.

Implemented data governance and MDM solutions using Collibra, Erwin, Informatica MDM, and Ataccama, supporting data lineage, cataloging, metadata management, and data quality.

Developed data anomaly detection and validation frameworks using PySpark and Python, implementing schema evolution and compatibility checks to improve data reliability.

Built predictive and descriptive analytics solutions using Python, PySpark, Databricks, MLflow, Scikit-learn, TensorFlow, NLTK, and spaCy.

Leveraged AWS S3, EC2, EMR, Lambda, Redshift, IAM, CloudFormation and GCP BigQuery/Dataflow for cloud data processing, migration, and analytics.

Developed large-scale ingestion and migration pipelines using Apache NiFi, Sqoop, Kafka, HBase, Cassandra, MongoDB, and Amazon DocumentDB.

Implemented workflow orchestration using Apache Airflow, Oozie, Shell scripting, and Python, automating data processing, scheduling, monitoring, and dependency management.

Developed Power BI dashboards and reports for KPI tracking and business intelligence, integrating data from enterprise and external systems including Salesforce.

Implemented CI/CD and DevOps practices using Jenkins, GitHub Actions, Docker, Kubernetes, and Git, and worked in Agile/Scrum environments.

Environment: Microsoft Fabric, Azure Data Factory, Azure Databricks, Apache Spark, PySpark, Scala, Kafka, Hadoop, HDFS, Hive, Snowflake, Snowpark, AWS, GCP BigQuery, Dataflow, Power BI, Python, SQL, Informatica, Pentaho, Collibra, Erwin, ER/Studio, Airflow, NiFi, Cassandra, MongoDB, Docker, Kubernetes, Jenkins, Azure DevOps, GitHub Actions

Blue5green, India

Sep 2020 – Oct 2022

Role: Data Engineer

Responsibilities:

Designed and developed scalable data ingestion and processing pipelines using Apache Kafka, Apache Spark, PySpark, Hive, Python, and SQL to process high-volume transactional and event data.

Developed real-time and batch data processing workflows using Spark Streaming and PySpark, implementing data transformation, validation, enrichment, partitioning, and performance optimization.

Built enterprise ETL/ELT workflows using Ab Initio, Informatica, Hive, and SQL, supporting data extraction, transformation, validation, reconciliation, and delivery to analytical platforms.

Implemented data quality and reconciliation frameworks using Python and SQL to identify duplicates, missing values, data inconsistencies, and source-to-target mismatches.

Prepared large-scale datasets for Machine Learning and predictive analytics, performing data cleansing, feature engineering, transformation, and model-ready dataset preparation for fraud detection and behavioral analytics.

Designed dimensional data models using Star and Snowflake schemas, implementing SCD Type 1 and Type 2, fact and dimension tables, and historical reporting structures.

Developed and optimized Hive-based data processing solutions, implementing partitioning, bucketing, UDF/UDAF, and query optimization for large-scale analytical workloads.

Designed logical and physical data models using Erwin, ER/Studio, and SQL Server, including normalization, dimensional modeling, reverse engineering, data dictionaries, and metadata management.

Architected and implemented AWS Redshift data warehouse solutions, developing data models, loading strategies, and performance optimization techniques for enterprise analytics.

Built Python-based automation and data processing frameworks using Pandas and NumPy for data profiling, cleansing, validation, reconciliation, and reporting.

Developed scalable Python microservices using Django and Flask, exposing data-processing capabilities through REST APIs and integrating enterprise applications with data platforms.

Implemented CI/CD pipelines using Jenkins, Git, Maven, and automated testing frameworks to support consistent deployments across development, QA, and production environments.

Built monitoring and observability solutions using Grafana, Kibana, and Prometheus to monitor data pipelines, application health, processing failures, and system performance.

Automated reporting and batch-processing workflows using UNIX/Linux Shell scripting and Teradata utilities, including BTEQ, FastLoad, and MultiLoad.

Participated in the complete SDLC, including requirements analysis, data modeling, development, testing, deployment, documentation, performance tuning, and production support, while collaborating with DBAs, business stakeholders, and cross-functional teams.

Environment: Python, PySpark, Apache Spark, Spark Streaming, Kafka, Hive, SQL, Ab Initio, Informatica, AWS Redshift, Oracle 12c, SQL Server, Teradata, PL/SQL, Erwin, ER/Studio, Pandas, NumPy, Power BI, Jenkins, Git, Maven, Grafana, Kibana, Prometheus, Django, Flask, REST APIs, UNIX, Linux, Shell Scripting, XML, Selenium, Java, TestNG

Amazon, India

Oct 2018 – Aug 2020

Role: Data Analyst / Data Engineer

Responsibilities:

Developed scalable Kafka-based streaming solutions and Apache Spark ETL pipelines to process real-time and multi-terabyte transactional datasets across Hadoop and cloud platforms.

Built machine learning training datasets and analytical solutions for fraud detection and predictive analytics, using Python, R, Pandas, NumPy, and statistical techniques.

Designed dimensional data models, including fact and dimension tables, supporting enterprise reporting, analytics, and business intelligence.

Implemented data cleansing, validation, reconciliation, and balancing frameworks using Python and SQL, improving data accuracy and reliability.

Designed event-driven ETL orchestration using AWS Lambda and Step Functions, implementing workflow dependencies, retries, error handling, and automation.

Ingested data from Oracle, SQL Server, and Teradata into HDFS using Sqoop, and developed Hive tables and HiveQL transformations for large-scale distributed processing.

Installed and configured Apache Airflow, integrating AWS S3 and Snowflake as primary data sources and targets for automated data pipelines.

Optimized Snowflake virtual warehouses using zero-copy cloning, Time Travel, and workload optimization, achieving approximately 40% cost savings without impacting performance.

Applied pushdown optimization techniques to improve performance of Informatica BDE pipelines and optimized large-scale ETL workloads.

Developed Power BI dashboards and reports, integrating data from SQL Server and Azure SQL and modernizing legacy reporting solutions.

Implemented CI/CD pipelines using Jenkins and automated ETL workflows using AutoSys, cron, Shell scripting, and Informatica jobs.

Built monitoring and metrics pipelines using Query Exporter and Prometheus, supporting operational monitoring and performance analysis.

Environment: Python, R, SQL, Apache Spark, Kafka, Hadoop, HDFS, Hive, HiveQL, Sqoop, Snowflake, AWS S3, Lambda, Step Functions, Airflow, Informatica BDE, Jenkins, Power BI, Azure SQL, Oracle, SQL Server, Teradata, Pandas, NumPy, Django, Flask, AutoSys, Shell Scripting, Prometheus

Value Labs, India

Jul 2016 – Sep 2018

Role: Data Analyst

Responsibilities:

Gathered business requirements and translated them into analytical reports, dashboards, KPIs, and data solutions in collaboration with business stakeholders.

Developed complex SQL queries, stored procedures, views, and data extraction processes to support reporting and business analytics.

Performed data cleansing, validation, reconciliation, data mapping, and gap analysis, supporting enterprise data warehouse and ETL initiatives.

Created MIS reports, OLAP solutions, cubes, scorecards, dashboards, and KPI reports using Tableau and Excel.

Automated recurring reporting processes and data preparation workflows, reducing manual effort and reporting time.

Supported ETL and data warehouse teams in identifying and resolving data quality issues across OLTP and OLAP systems.

Used Teradata utilities, including FastExport and MultiLoad (MLOAD), for data migration and ETL processing between OLTP source systems and OLAP target environments.

Developed knowledge of Hadoop architecture and ecosystem components, including HDFS, NameNode, DataNode, Secondary NameNode, JobTracker, TaskTracker, and MapReduce.

Participated in UAT, report validation, testing, documentation, and data dictionary maintenance throughout the SDLC.

Worked in Agile methodology, collaborating with cross-functional teams to deliver reporting and analytical solutions.

Environment: SQL, Teradata, FastExport, MLOAD, OLAP, OLTP, Hadoop, HDFS, MapReduce, Tableau, Excel, ETL, Data Warehousing, Data Validation, Agile

EDUCATION:

University of the Pacific, Stockton, CA

Master of Computer Science 2025

Contact this candidate