Data Engineer Processing

BHARGAVI RAMBATHRI

DATA ENGINEERING

M: +1-475-***-**** E: **************@*****.*** L: LinkedIn SUMMARY

• Data Engineer with over 5+ years of experience in SQL, Python, AWS, Azure, Spark, and Hadoop, delivering high-accuracy data solutions and optimizing processing times for large-scale businesses.

• Proven expertise in enhancing data pipeline reliability by leveraging Airflow's error handling and alerting features, minimizing disruptions in data processing and ensuring consistent data quality.

• Specialized in Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats, transforming data to uncover valuable insights into customer usage patterns.

• Extensive experience in Database design, Data Modeling, Data Migration, Data Cleansing, and ETL processes, with a deep understanding of both RDBMS (including SQL Server, MySQL) and NoSQL (such as MongoDB, HBase) technologies, enabling the design and implementation of tailored solutions for diverse data needs. EXPERIENCE

DATA ENGINEERING Capital One VA - USA JAN 2023 – PRESENT

• Established and maintained an ETL pipeline using Informatica PowerCenter to extract, transform, and load data from multiple sources into a data warehouse, ensuring data accuracy and consistency.

• Developed a complex data processing workflow using AWS Pipeline, involving multiple stages (data extraction, transformation, loading), which improved data pipeline reliability and resulted in a 40% reduction in data processing errors.

• Optimized Lambda functions for cost efficiency, achieving a 20% reduction in execution costs through code refactoring and best practices.

• Migrated the data aggregation layer from legacy services to Snowflake using Data Build Tool (DBT) models, resulting in up to 70% cost savings and improved query performance.

• Orchestrated complex data pipelines with 3 stages using AWS Step Functions to automate data movement and transformation tasks, ensuring reliable data flow.

• Implemented a data warehouse on Databricks using Delta Lake, improving data query performance by 40% compared to the previous Teradata-based solution.

DATA ENGINEERING Dixon Technology India JUN 2019 - JUL 2022

• Implemented AWS Data Pipeline to orchestrate complex data workflows, reducing manual intervention by 30% and enhancing data reliability.

• Developed Apache Flink streaming applications to handle real-time data processing with low latency, achieving 20% faster results compared to Apache Spark Streaming.

• Automated data ingestion and transformation pipelines using AWS Glue and AWS Pipeline, minimizing manual effort and significantly improving data processing speed.

• Designed and deployed a high-performing ETL pipeline in Databricks to process terabytes of data daily, facilitating seamless data integration and transformation for the data warehouse.

• Built data integration pipelines using Informatica PowerCenter, automating data movement and reducing manual effort by 50%.

• Optimized Hive query execution time by 25% through techniques such as partitioning, bucketing, and cost-based optimization.

TECHNICAL SKILLS

Methodologies: SDLC, Agile/Scrum, Waterfall

Programming Language and IDE’s: Scala, Python, SQL, Visual Studio Code, PyCharm, Jupyter Notebook Big Data Ecosystem: Hadoop, MapReduce, Hive, Pig, DynamoDB, BigQuery, HDFS, Spark, HBase, Kafka Machine Learning: Linear Regression, Logistic Regression, Decision Tree, SVM, K mean, Random Forest Cloud Technologies: AWS (EC2, S3 Bucket, Redshift, Lambda, RDS, DynamoDB), Azure (Data Lake, Data Factory, Databricks)

Packages: NumPy, Pandas, Matplotlib, SciPy, Scikit-learn, Seaborn, TensorFlow, Kafka, PySpark ETL and Reporting Tools: Apache Spark, Apache Airflow, Tableau, Power BI, SSRS, SSIS, Informatica Database: MS SQL Server, PostgreSQL, MongoDB, MySQL, Cassandra, Snowflake Other Technical Skills: Data Modeling, Data Warehouse, Data Quality, Data Analysis, Data Governance, Team Collaboration EDUCATION

Master’s in computer science & information technologies SEP 2022- DEC 2023 Sacred Heart University, CT, USA Bachelor’s in computer science JUN 2018- MAY 2021 Prathibha Degree & PG College, Hyderabad, TS, India

Contact this candidate