Senior Data Engineer - 10+ Years in Scalable Data Platforms

Malvika Yedla Mallesh

+1-469-***-**** *********@*****.*** LinkedIn: https://www.linkedin.com/in/malvika-ym/

PROFESSIONAL SUMMARY

Senior Data Engineer with 10+ years of experience designing and building scalable data platforms using Big Data and Spark technologies across enterprise environments.

Engineered high-performance data pipelines using Spark, PySpark, Hadoop, Hive, Kafka, and Airflow to process large-scale structured and unstructured datasets.

Delivered data solutions across Healthcare, Insurance, and financial domains, leveraging Cloudera and Hortonworks distributions for enterprise-grade implementations.

Developed robust applications and data workflows using Python, Scala, Java, and SQL, enabling efficient data processing and automation.

Architected and deployed cloud-native data platforms on AWS, Azure, and GCP, building scalable and cost-efficient ETL/ELT pipelines.

Designed and implemented data warehouse solutions using Snowflake, Redshift, BigQuery, and Synapse, improving query performance and analytics capabilities.

Built and orchestrated end-to-end ETL pipelines using Airflow, Azure Data Factory, and Control-M, ensuring reliable and scalable data processing.

Implemented real-time and batch data processing systems using Kafka, Spark Streaming, and cloud-native streaming services.

Migrated on-premises data platforms to cloud environments (AWS, Azure, GCP), improving scalability, performance, and cost efficiency.

Designed dimensional data models (Star, Snowflake, SCD) to support high-performance analytics and reporting systems.

Developed and optimized data integration solutions using Informatica PowerCenter and ETL frameworks for enterprise data workflows.

Implemented data governance, lineage, and metadata management solutions using Collibra, Informatica, and Alteryx.

Built large-scale data processing applications using Hadoop ecosystem tools including HDFS, MapReduce, Hive, HBase, Flume, and Sqoop.

Leveraged Spark Core, Spark SQL, and Spark Streaming to process multi-terabyte datasets with improved performance using in-memory computing.

Designed and implemented ETL/ELT pipelines using Apache Beam and Cloud Dataflow for scalable data transformation.

Applied data security practices including data masking, anonymization, and compliance frameworks using Informatica Data Quality (IDQ).

Built and managed large-scale data warehouse solutions integrating Snowflake, Redshift, and relational databases for enterprise analytics.

Worked with NoSQL databases such as Cassandra, MongoDB, and HBase to support high-volume and low-latency data applications.

Optimized SQL Server databases through indexing, query tuning, and performance optimization techniques.

Developed complex SQL queries, stored procedures, and functions to support business intelligence and reporting requirements.

TECHNICAL SKILLS

Big Data Technologies

Hadoop, MapReduce, Spark, HDFS, Sqoop, YARN, Oozie, Hive, Impala, Zookeeper, Apache Flume, Apache Airflow, Cloudera, HBase

Programming Languages

Python, PL/SQL, SQL, Scala, C, C#, C++, T-SQL, Power Shell Scripting, JavaScript, Perl script.

Cloud Technologies

AWS, Microsoft Azure, GCP, Databricks, Snowflake

Cloud Services

Azure Data Lake Storage Gen 2, Azure Data Factory, Blob storage, Azure SQL DB, Databricks, Azure Event Hubs, AWS RDS, Amazon SQS, Amazon S3, AWS EMR, Lambda, AWS SNS, Data Flow, Big Query, VM, Delta Tables, Cloud functions, Clusters.

Databases

MySQL, SQL Server, IBM DB2, Postgres, Oracle, MS Access, Teradata, and Snowflake

NoSQL Data Bases

MongoDB, Cassandra DB, HBase

Development Strategies

Agile, Lean Agile, Pair Programming, Waterfall, and Test-Driven Development.

ETL, Visualization & Reporting

Tableau, Data Stage, Informatica, Talend, SSIS, and SSRS

Frameworks

Django, Pandas, NumPy, Matplot Lib, TensorFlow, PyTorch

Version Control & Containerization tools

Jenkins, Git, CircleCI and SVN

Operating Systems

Unix, Linux, Windows, Mac OS

Monitoring tool

Apache Airflow, Control M

Tools

PyCharm, Eclipse, Visual Studio, SQL*Plus, SQL Developer, TOAD, SQL Navigator, Query Analyzer, SQL Server Management Studio, SQL Assistance, Eclipse, Postman

Machine Learning Techniques:

Linear & Logistic Regression, Classification and Regression Trees, Random Forest, Associative rules, NLP and Clustering.

PROFESSIONAL EXPERIENCE

Wells Fargo, San Francisco, California March2024-Present

Sr. Data Engineer

Responsibilities:

●Designed end-to-end Azure data solutions using Data Factory, Databricks, ADLS Gen2, and Synapse Analytics to build comprehensive enterprise data platforms.

●Collaborated with business leaders to understand analytical requirements and delivered data solutions that directly connected insights to business strategy, enhancing reporting capabilities for investment analysis.

●Developed scalable data pipelines with Azure Data Factory orchestrating data movement and transformations across diverse data sources including databases, APIs, and file systems.

●Developed interactive dashboards using Databricks SQL Endpoints and native visualizations, enabling business stakeholders to self-serve analytics-ready datasets.

●Built optimized data processing workflows using Azure Databricks with Spark (Scala/Python) for data ingestion, cleansing, feature engineering, and advanced analytics.

●Monitored production systems to identify and resolve issues with TRIRIGA jobs, ensuring system stability and SLA compliance.

●Implemented real-time processing solutions with Azure Event Hubs, Stream Analytics, and Databricks Structured Streaming for low-latency data ingestion and analysis.

●Managed end-to-end lifecycle of Microsoft Fabric projects including design, deployment, and ongoing maintenance ensuring scalability, dependability, and high availability.

●Led data quality initiatives using Informatica Data Quality (IDQ) to ensure accuracy, consistency, and reliability of critical data assets across the organization.

●Designed conceptual, logical, and physical data models for enterprise data platforms using Azure Synapse and Databricks Delta Lake.

●Developed dimensional models (Star Schema) to support business intelligence and analytics reporting for investment analysis.

●Implemented slowly changing dimensions (SCD Type 1 & 2) to maintain historical data accuracy and traceability.

●Tuned Spark shuffle configurations including shuffle partitions, memory buffer sizes, and serialization formats for improved performance and resource utilization.

●Designed message queue solutions using Azure Service Bus to enable reliable and asynchronous communication between distributed systems and applications.

●Developed data warehousing solutions on Azure Synapse including data modeling, SQL script development, and query optimization for high-performance analytics.

●Implemented machine learning workflows using Azure Databricks and Azure ML including data preparation, model training, and production deployment.

●Configured event-driven architectures with Azure Event Grid for reactive programming, including event filtering rules and routing logic to appropriate handlers.

●Integrated Azure Cosmos DB for globally distributed, low-latency applications leveraging its multi-model capabilities for diverse use cases.

●Troubleshot and resolved TRIRIGA production issues including job failures, workflow malfunctions, and data inconsistencies through in-depth review of logs and integration components.

●Implemented data security measures including role-based access control (RBAC), encryption at rest and in transit, and data masking techniques ensuring compliance.

●Utilized Azure Monitor, Log Analytics, and Application Insights for observability, proactive troubleshooting, and performance optimization of data pipelines.

●Managed Databricks clusters including provisioning, monitoring, auto-scaling, and auto-termination policies to optimize resource utilization and cost management.

●Developed .NET applications in C# for data integration, transformation, and processing from various sources including databases, APIs, and streaming platforms.

●Collaborated with cross-functional teams to gather requirements, design data solutions, and deliver projects on time and within budget.

●Implemented data partitioning and optimization techniques in Azure Databricks to improve processing performance and reduce operational costs.

●Built CI/CD pipelines using Azure DevOps to automate deployment of data pipelines and workflows ensuring consistent and reliable releases.

●Implemented data lineage and metadata management solutions to track and document data transformations using Azure services and Databricks Delta Lake.

●Optimized Spark jobs and SQL queries in Azure Databricks for improved performance, resource utilization, and cost efficiency.

●Monitored and logged tickets in JIRA/ServiceNow to meet resolution SLAs for TRIRIGA production issues.

●Established data archival and retention policies using Azure Blob Storage and ADLS ensuring long-term data compliance and lifecycle management.

●Deployed machine learning models into production using Azure ML, integrating them into data workflows and Databricks pipelines for real-time predictions.

●Troubleshot complex issues across Azure services, Databricks clusters, and data pipelines identifying root causes and implementing resolutions.

●Performed root cause analysis of TRIRIGA process issues and implemented permanent fixes to prevent recurrence.

●Created Python scripts to transform, aggregate, and prepare data for dashboard consumption in Databricks and Tableau, reducing manual preparation time by 40%.

●Implemented data cataloging and metadata management using Azure Data Catalog and Databricks Delta Lake for improved data discovery and governance.

●Implemented Medallion architecture (Bronze Silver Gold layers) using Databricks Delta Lake and Auto Loader for curated, analyticsready datasets.

●Secured credentials and secrets using Azure Key Vault, integrated with Databricks and ADF pipelines.

●Created visualizations and insights using Power BI, Azure Synapse Studio, and Azure Data Explorer enabling data-driven decision-making for stakeholders.

●Implemented Azure Active Directory for authentication and authorization in .NET applications enabling secure access control and user management.

●Integrated Azure Cognitive Services with data pipelines for text analytics, image recognition, and natural language processing enhancing data capabilities.

●Designed Logic Apps workflows for data transformation, cleansing, enrichment, and normalization preparing data for downstream analytics and consumption.

●Designed and implemented data migrations into IBM TRIRIGA from source systems including Maximo, SAP, and legacy IWMS platforms.

●Integrated Teradata and other enterprise sources into modern Azure pipelines ensuring seamless data movement from legacy systems.

●Implemented data access controls and auditing mechanisms to ensure data governance and compliance with regulatory requirements including GDPR and HIPAA.

Environment: Azure SQL, Azure Storage Explorer, Azure Storage, Azure Blob Storage, Azure Backup, Azure Files, Azure Data Lake Storage, SQL Server Management Studio 2016, Teradata, Visual Studio 2015, VSTS, Azure Blob, Power BI, PowerShell, C# .Net, SSIS, DataGrid, ETL Extract Transformation and Load, Business Intelligence (BI).

Datavant, Phoenix, Arizona Aug 2022-March2024 Sr. Data Engineer

Responsibilities:

Developed ETL workflows using SQL and scripting to move data from external sources into TRIRIGA staging tables.

Responsible for the execution of big data analytics, predictive analytics, and machine learning initiatives.

Expanded self-service data capabilities by building curated data marts and implementing data catalog solutions, enabling business teams to access and analyze data independently through Tableau and Snowflake.

Implemented a proof of concept deploying this product in AWS S3 bucket and Snowflake.

Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats.

Worked extensively with healthcare payer data domains including member enrollment, provider networks, and claims data to support predictive analytics and machine learning initiatives.

Used Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra.

Established and provisioned various Databricks clusters tailored for batch and continuous streaming data processing, inclusive of the installation of essential libraries. Integrated Azure Active Directory authentication into every Cosmos DB request, and effectively demonstrated this feature to stakeholders.

Written AWS Lambda code in Python for nested JSON files, converting, comparing, sorting etc.

Created a snowflake warehouse strategy and set it up to use PUT scripts to migrate a terabyte of data from S3 into Snowflake.

Strong expertise in developing and scheduling ETL workflows using AWS Glue jobs to automate data extraction, transformation, and loading processes.

Extensive experience in designing, implementing, and managing data warehousing solutions using Amazon Redshift.

Configured TRIRIGA job schedulers for ETL items including meter data loading and fact table processing.

Architected Data Vault 2.0 data models to ensure flexibility, scalability, and adaptability for healthcare payer data domains.

Designed and maintained Snowflake data warehouse models including fact and dimension tables for member enrollment, provider networks, and claims data.

Designed and implemented a Snowflake data warehouse architecture to support efficient data storage, retrieval, and analysis.

Implemented Data Vault methodology for data modeling, ensuring flexibility, scalability, and easy adaptability to changing business requirements.

Designed and implemented a robust data quality framework using Snowflake's built-in features and DBT (Data Build Tool) to perform data profiling, validation, and cleansing.

Created a data pipeline involving various AWS services including S3, Kinesis firehose, and kinesis data stream, SNS, SQS, Athena, and Snowflake etc.

Worked on end-to-end deployment of the project that involved Data Analysis, Data Pipelining, Data Modelling, Data Reporting and Data documentations as per the business needs.

Skilled in visualizing and presenting data using Tableau, creating interactive dashboards, and generating meaningful insights for stakeholders.

Designed and implemented data visualizations and charts in Tableau to effectively communicate complex data insights and trends to non-technical users.

Work closely with the application customers to resolve JIRA tickets related to API issues, data issues, consumption latencies, onboarding, and publishing data.

Environment: HDFS, Oozie, Spark, Kafka, Zookeeper, Python, HBase, Flume, AWS (EMR, S3, EC2, VPC, LAMBDA, EBS, CloudFront, IAM, CloudWatch, Redshift, DynamoDB), Teradata, DBT, Kafka, PySpark, Snowflake, Tableau.

CFA Institute, Charlottesville, Virginia June2020-July2022

Data Engineer

Responsibilities:

Designed and managed AWS-based data solutions using S3, Redshift, Glue, and EMR for enterprise analytics workloads.

Built scalable ETL pipelines using AWS Data Pipeline, AWS Glue, and Databricks to ingest and transform data from multiple sources.

Implemented Databricks and Spark workflows for data ingestion, transformation, and analytics on large datasets.

Developed Spark jobs in Scala and Python and used Databricks notebooks for exploration, engineering, and testing.

Managed Spark clusters on AWS Databricks including provisioning, configuration, and performance optimization.

Defined DBT models to represent business logic and create maintainable transformation layers on raw data.

Designed database schemas and tables using data modeling principles and Snowflake schema evolution capabilities.

Implemented real-time streaming architectures using Amazon Kinesis and Databricks Structured Streaming.

Wrote DBT tests and Snowflake stored procedures to improve data quality and automate orchestration tasks.

Implemented governance, security, archival, and monitoring practices across AWS using IAM, S3, Glue, CloudWatch, and related services.

Environment: Agile Scrum, Spark, Scala, Hive, Kafka, Python, AWS (EC2, S3, EBS, ELB, RDS, SNS, SQS, VPC, Cloud formation, CloudWatch, ELK Stack), Bitbucket, Ansible.

Freddie Mac, Tyson VA Nov2017-May 2020

Data Engineer

Responsibilities:

Designed configurable ETL frameworks in Python with YAML-based parameterization for intake, archival, reconciliation, and notifications.

Built data validation, reconciliation, and archival processes for mortgage and prepayment datasets with strong compliance controls.

Containerized Python jobs using Docker and deployed them on AWS EKS for scalable and resilient production execution.

Implemented CI/CD pipelines with Git, branching, pull requests, and automated deployment practices across environments.

Scheduled and orchestrated jobs in Control-M with dependency handling, alerts, and SLA-focused execution planning.

Integrated Informatica MDM to support master data governance and seamless data exchange with downstream systems.

Performed unit testing and validation of Python jobs to match legacy Data flux behavior and avoid data loss.

Developed SQL queries and Python scripts for validation, reconciliation, and downstream reporting across core data domains.

Built environment-driven YAML parameterization for reusable deployment across dev, test, and prod systems.

Collaborated with DevOps, QA, and business stakeholders to align migration execution with governance standards and production requirements.

Environment: SQL, Informatica, MDM, ETL, CRM, Data Quality, Data Governance, Informatica Power Center.

Logic Matter Inc. Hyderabad, India June 2015-Aug2017

Data Analyst

Responsibilities:

Designed and developed SSIS-based ETL solutions to load data into SQL Server databases, data marts, and data warehouses.

Created complex transformations and control-flow logic in SSIS to support accurate and efficient data processing.

Implemented incremental load strategies to process large datasets and reduce ETL execution windows.

Developed interactive SSRS dashboards and reports with drill-down, drill-through, and parameterized functionality.

Optimized SSIS package performance through buffer tuning, transformation optimization, and parallel execution strategies.

Managed deployment and scheduling of SSIS packages using SQL Server Agent across multiple environments.

Implemented data validation and cleansing techniques using SQL Server data quality capabilities and custom transformations.

Configured SSIS connection managers for flat files, relational systems, OLE DB sources, and web service integrations.

Developed custom SSIS components using Script tasks and Script transformations for specialized processing needs.

Collaborated with business stakeholders to gather reporting requirements and deliver production-ready reporting solutions

Environment: SQL, SSIS, SSRS, ETL, SQL Server, Data Warehouse, Data Mart.

EDUCATIONAL DETAILS

JNTUH Hyderabad, India Aug 2010- May 2014 BTech.

Contact this candidate