Data Engineer Senior

Executinve Summary

Over **+ years of IT industry experience which includes 07 years of Senior Data Engineer experience in Big Data, Hadoop, Spark, Data Visualizations using Tableau, writing queries, processing, and analysis of large-scale data. Solid experience in Hadoop ecosystem & technologies including HDFS, Map Reduce, Hive, Sqoop, PySpark, Oozie and Hue as well as Data Warehouse & Reports’ testing. Utilized high quality software applications, best Project Management, quality practices & processes.

Technical Skills:

Big Data Technologies

Hadoop, Hive, Sqoop, Oozie, PySpark

Databases

Oracle, Teradata

Programming Languages / Scripting

Shell, Python, R

Business Intelligence

Tableau

Web Analytics Tool

Google Analytics

Tools & Utilities

Autosys, D-series, Jenkins, Jira

Work Experience:

Employer: Infosys

Client: Bank of America

Role: Data Engineering Technology Lead (Aug 2021 – Present)

Responsibilities:

Designed & developed data pipelines to ingest various types of data files such as text, csv, structured and semi structured XML to Hadoop, for applications such as Purple Census, CMX Request and Responses & Prepaid MRA Harm etc., by using Map Reduce/Spark.

Designed & developed ten CI/CD deployment pipelines by using Jenkins to deploy the packaged data pipelines.

Collaborating with cross-functional teams, key stakeholders, and Lines of Business (LOBs) to perform requirements and data analysis as well as update the project status.

Writing queries based on business requirements and loading data to Hive tables.

Extracting data from Hive tables to flat files and exporting the files to external clients via SFTP.

Mentoring new members on the team.

Performing data analysis using Hive and providing Ad hoc reports to LOBs.

Performed data and application migration validations from RHEL6 to RHEL 7.

Certified successful completion of data processing and transfers within the required time constraints.

Employer: VISA, Inc

Role: Senior Data Engineer

Responsibilities:

As a member of the Data Platform team, implemented data pipelines for various products such as Visa Checkout, Visa Direct etc. by using Sqoop.

Extracted data from various types of data sources such as Oracle, csv etc. by using Sqoop Import to ingest into Hive tables.

Created and maintained optimal data pipelines’ design for applications.

Collaborated with DevOps team to perform production deployments and provided production support to fix the defects.

Used Hive extensively to perform transformations and pre-aggregations for developing baseline data for reports.

Implemented Oozie workflow engine to run Hive jobs.

Migrated Hive jobs to PySpark jobs using D-series automated tool.

Loaded data into Hive tables with static and dynamic partitions, intended for efficiency.

Worked on various Hadoop file formats such as Parquet, ORC, AVRO, Text & CSV.

Configured Encryption Framework in Production & non-production environments and tested encryption/decryption capabilities for Visa Checkout application.

Produced attractive visuals/dashboards to convey the story inside the data.

Designed, developed, tested, and maintained Tableau functional reports based on business requirements.

Provided production support to Tableau users and wrote custom SQL to support business requirements.

Gathered and analyzed business requirements, groomed stories, and features at an early stage, and built solutions using various Big Data technologies to address the business capabilities.

Played the role of Program Manager for the project.

Performed Functionality, Integration, System, Regression, ETL and Reports’ testing.

Educational Qualifications:

Master of Computer Applications (MCA), Visvesvaraya Technological University, Belgaum, Karnataka, India.

Bachelore of Science (B.Sc.), Kuvempu University, Karnataka, India.

Professional Certification(s):

Data Analytics Certificate Program, University of Texas at Austin, TX.

Employment History:

Employer

Designation

From

To

On-site Experience

Infosys, Ltd

Data Engineering Technology Lead

August, 2021

May,2024

Visa, Inc

Sr. Data Engineer

August, 2016

August, 2021

Wipro

Test Lead

January, 2012

June, 2014

ORACLE

Team Lead

June, 2010

January, 2012

Capgemini Consulting Ltd

Consultant

April, 2007

June, 2010

Perot Systems

Test Engineer

July, 2004

April, 2007

Worked in London, UK for the client, UBS, for 06 months from March, 2006 through September, 2006.

Contact this candidate