Data Engineer

Name: Srihari B

Email ID: ***********@*****.*** Phone Number: +1-571-***-****

* ***** ** ** ********** in a variety of industries working on Big Data technology using technologies such as Cloudera and Hortonworks distributions. Hadoop working environment includes Hadoop, Spark, MapReduce, Kafka, Hive, Ambari, Sqoop, HBase, and Impala.

Have Extensive Experience in IT data analytics projects, Hands on experience in migrating on premise ETLS to Google Cloud Platform (GCP) using cloud native tools such as BIG query, Cloud Data Proc, Google Cloud Storage, Composer

Fluent programming experience with Scala, Java, Python, SQL, T-SQL, R.

Hands-on experience in developing and deploying enterprise-based applications using major Hadoop ecosystem components like MapReduce, YARN, Hive, HBase, Flume, Sqoop, Spark MLIib, Spark GraphX, Spark SQL, Kafka.

Adept at configuring and installing Hadoop/Spark Ecosystem Components.

Proficient with Spark Core, Spark SQL. Spark MLIib, Spark GraphX and Spark Streaming for processing and transforming complex data sing in-memory computing capabilities written in Scala. Worked with Spark to improve efficiency of existing algorithms using Spark Context, Spark SQL, Spark MLIib, Data Frame, Pair RDD's and Spark YARN.

Experience in application of various data sources like Oracle SEZ, SQL Server, Flat Files and Unstructured files into a data warehouse.

Able to use Sqoop to migrate data between RDBMS, NoSQL databases and HDFS.

Experience in Extraction, Transformation and Loading (ETL) data from various sources into Data Warehouses, as well as data processing like collecting, aggregating and moving data from various sources using Apache Flume, Kafka, Powers! And Microsoft SSIs.

Hands-on experience with Hadoop architecture and various components such as Hadoop File System HDF, Job Tracker, Task Tracker, Name Node, Data Node and Hadoop MapReduce programming.

Comprehensive experience in developing simple to complex Map reduce and Streaming jobs using Scala and Java for data cleansing, filtering and data aggregation. Also possess detailed knowledge of MapReduce framework.

Used IDEs like Eclipse, Intell IDE, PyCharm IDE, Notepad ++, and Visual Studio for development.

Seasoned practice in Machine Learning algorithms and Predictive Modeling such as Linear Regression, Logistic Regression, Naive Bayes, Decision Tree, Random Forest, KNN, Neural Networks, and K-means Clustering.

Ample knowledge of data architecture including data ingestion pipeline design, Hadoop/Spark architecture, data modeling, data mining, machine learning and advanced data processing.

Experience working with NoSQL databases like Cassandra and HBase and developed real-time read/write access to very large datasets via HBase.

Developed Spark Applications that can handle data from various RDBMS (MySQL Oracle Database) and Streaming sources.

Proficient SQL experience in querying, data extraction/transformations and developing queries for a wide range of applications.

Capable of processing large sets (Gigabytes) of structured, semi-structured or unstructured data.

Experience in analyzing data using HiveQL. Pig, HBase and custom MapReduce programs in Java 8.

Experience working with GitHUb/Git 2.12 source and version control systems.

Strong in core Java concepts including Object-Oriented Design (OOD) and Java components like Collections Framework, Exception handling, I/0 system.

Technical Skills:

Hadoop

Hadoop, Spark(Pyspark),Map Reduce, HIVE, PIG, Impala SQOOP, HDFS, HBASE, Oozie, Ambari, Spark, Scala and Mongo DB

Cloud Technologies

AWS Kinesis, Lambda, EMR, EC2, SNS, SQS, Dynamo DB, Step Functions, Glue, Athena, CloudWatch, Azure Data Factory, Azure Data Lake, Functions, Azure SQL Data Warehouse, Databricks and HDInsight

DBMS

Amazon Redshift,Postgres, Oracle 9i, SQL Server, IBM DB2 And TeraData

ETL Tools

Data Stage, Talend and ABInitio

Reporting Tools

Power BI, Tableau, TIBCO Spotfire, Qlikview and Qliksense

Deployment Tools

Git, Jenkins, Terraform and CloudFormation

Programming Language

Python, Scala, PL/SQL and Java

Scripting

Unix Shell and Bash scripting

Client Name: Amazon Web Services, Herndon VA Aug 2022 – Till Date

Role: Sr. Data Engineer

Responsibilities:

Worked on AWS Data pipeline to configure data loads from S3 to into Redshift.

Using AWS Redshift, Extracted, transformed and loaded data from various heterogeneous data sources and destinations.

Created Tables, Stored Procedures, and extracted data using T-SQL for business users whenever required.

performs data analysis and design, and creates and maintains large, complex logical and physical data models, and metadata repositories using ERWIN and MB MDR

I have written shell script to trigger data Stage jobs.

Assist service developers in finding relevant content in the existing reference models.

Like Access, Excel, CSV, Oracle, lat files using connectors, tasks and transformations provided by AWS Data Pipeline.

Utilized Spark SQL AP in Pyspark to extract and load data and perform SQL queries.

Worked on developing Pyspark script to encrypting the raw data by using hashing algorithms concepts on client specified columns.

Responsible for Design, Development. and testing of the database and Developed Stored Procedures, Views, and Triggers

Developed Python-based API (RESTful Web Service) to track revenue and perform revenue analysis.

Compiling and validating data from all departments and Presenting to Director Operation.

KPI calculator Sheet and maintain that sheet within SharePoint.

Created Tableau reports with complex calculations and worked on Ad-hoc reporting using PowerBI.

Creating data model that correlates all the metrics and gives a valuable output.

Worked on the tuning of SQL Queries to bring down run time by working on Indexes and Execution Plan.

Performing ETL testing activities like running the Jobs, Extracting the data using necessary queries from database transform and upload into the Data warehouse servers.

Pre-processing using Hive and Pig.

Developed a detailed project plan and helped manage the data conversion migration from the legacy system to the target snowflake database.

Design, develop, and test dimensional data models using Star and Snowflake schema methodologies under the Kimball method.

Developed a detailed project plan and helped manage the data conversion migration from the legacy system to the target snowflake database.

Design, develop, and test dimensional data models using Star and Snowflake schema methodologies under the Kimball method.

Developed data pipeline using Spark, Hive, Pig, python, Impala, and HBase to ingest customer

Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.

Ensure deliverables (Daily, Weekly & Monthly MIS Reports) are prepared to satisfy the project requirements cost and schedule

Worked on a direct query using PowerBI to compare legacy data with the current data and generated reports and stored and dashboards.

Designed SSS Packages to extract, transfer, load (ETL) existing data into SQL Server from different environments for the SSAS cubes (OLAP)

SQL Server reporting services (SSRS). Created & formatted Cross-Tab, Conditional, Drill-down, Top N, Summary, Form, OLAP, Subreports, ad-hoc reports, parameterized reports, interactive reports & custom reports

Created action filters, parameters and calculated sets for preparing dashboards and worksheets using Powers!

Developed visualizations and dashboards using Powers!

Used ETL to implement the Slowly Changing Transformation, to maintain Historically Data in Data warehouse.

Performing ETL testing activities like running the jobs, Extracting the data using necessary queries from database transform, and upload into the Data warehouse servers.

Created dashboards for analyzing POS data using Power BI

Environment: MS SQL Server 2016, T-SQL, SQL Server Integration Services (SSIS). SQL Server Reporting Services (SSRS), SQL Server Analysis Services (SSAS). Management Studio (SSMS), Advance Excel (creating formulas, pivot tables, Hlookup, Viookup, Macros). Spark, Python, ETL. Power BI, Tableau, Hive/Hadoop. Snowflakes, Power Bl, AWS Data Pipeline, IBM Cognos 10.1, Data Stage. Cognos Report Studio 10.1, Cognos 8 & 10 BI, Cognos Connection, Cognos office Connection, Cognos 8.2/3/4, Data stage and Quality Stage 7.5

Client Name: Wayfair, Boston MA Dec 2019 – July 2022

Role: GCP Data Engineer

Responsibilities:

Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in GCP

strong understanding of AWS components such as EC2 and S3

Can work parallelly in both GCP and Azure Clouds coherently

Experience in moving data between GCP and Azure using Azure Data Factory.

Implemented a Continuous Delivery pipeline with Docker and Git Hub

Worked with g-cloud function with Python to load Data in to Bigquery for on arrival csv files in GCS bucket

Process and load bound and unbound Data from Google pubsub topic to Bigquery using cloud Dataflow with Python.

Devised simple and complex SQL scripts to check and validate Dataflow in various applications.

Performed Data Analysis, Data Migration, Data Cleansing, Transformation, Integration, Data Import, and Data Export through Python.

Developed and deployed data pipeline in cloud such as AWS and GCP

Performed data engineering functions: data extract, transformation, loading, and integration in support of enterprise data infrastructures - data warehouse, operational data stores and master data management

Responsible for data services and data movement infrastructures good experience with ETL concepts, building ETL solutions and Data modeling

Architected several DAGs (Directed Acyclic Graph) for automating ETL pipelines

Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data

Factory, T-SQL. Spark SQL, and U-SQL Azure Data Lake Analytics.

Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.

Implement ad-hoc analysis solutions using Azure Data Lake Analytics/Store, HDInsight

Implemented Copy activity, Custom Azure Data Factory Pipeline Activities

Primarily involved in Data Migration using SQL. SQL Azure, Azure Storage, and Azure Data Factory, SSIS, PowerShell.

Architect & implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB).

Migration of on-premise data (Oracle/ SQL Server/ DB2/ MongoDE) to Azure Data Lake and Stored (ADLS) using Azure Data

Hands on experience on architecting the ETL transformation layers and writing spark jobs to do the processing.

Gather and process raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, writing applications)

Experience in fact dimensional modeling (Star schema, Snowflake schema), transactional modeling and SCD (slowly changing dimension)

Devised PL/SQL Stored Procedures, Functions, Triggers, Views and packages. Made use of Indexing, Aggregation and Materialized views to optimize query performance.

Developed logistic regression models (Python) to predict subscription response rate based on customers variables like past transactions, response to prior mailings, promotions, demographics, interests, and hobbies, etc.

Develop near real time data pipeline using spark

Process and load bound and unbound Data from Google pub/sub topic to Bigquery using cloud Dataflow with Python

Hands of experience in GCP, Big Query, GCS bucket, G - cloud function, cloud dataflow, Pub/suB cloud shell, GSUTIL, BQ command line utilities, Data Proc, Stack driver

Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines

Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling

Worked on confluence and Jira skilled in data visualization like Matplotlib and seaborn library

Hands on experience with big data tools like Hadoop, Spark, Hive

Experience implementing machine learning back-end pipeline with Pandas, Numpy

Environment: Gcp, Big query. Ges Bucket, G-Cloud Function, Apache Beam, Cloud Dataflow, Cloud Shell, Gsutil, Docker, Kubernetes, AWS, Apache Airflow, Python, Pandas, Matplotlib, seaborn library, text mining, Numpy, Scikit-learn, Heat maps, Bar charts, Line charts, ETL workflows, linear regression, multivariate regression, Python, Scala, Spark Spark Developer

Client Name: MicroInfo, Bangalore, IND Aug 2017 - Sep 2019

Role: Data Engineer

Responsibilities:

Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines

Designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines

Performed data extraction, transformation, loading, and integration in data warehouse, operational data stores and master data management

Strong understanding of AWS components such as EC2 and $3

Performed Data Migration to GCP

Responsible for data services and data movement infrastructures

Experienced in ETL concepts, building ETL solutions and Data modeling

Worked on architecting the ETL transformation layers and writing spark jobs to do the processing.

Aggregated daily sales team updates to send report to executives and to organize jobs running on Spark clusters.

Loaded application analytics data into data warehouse in regular intervals of time

Designed & build infrastructure for the Google Cloud environment from scratch

Experienced in fact dimensional modeling (star schema, Snowflake schema), transactional modeling and SCD (Slowly changing dimension)

Leveraged cloud and GPU computing technologies for automated machine learning and analytics pipelines, such as AWS, GCP

Worked on confluence and Jira

Designed and implemented configurable data delivery pipeline for scheduled updates to customer facing data stores built with Python

Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling

Compiled data from various sources to perform complex analysis for actionable results

Measured Efficiency of Hadoop/Hive environment ensuring SLA is met

Optimized the Tensorflow Model for efficiency

Analyzed the system for new enhancements/functionalities and perform Impact analysis of the application for implementing ETL changes

Implemented a Continuous Delivery pipeline with Docker, and Git Hub and AWS.

Built performant, scalable ETL processes to load, cleanse and validate data

Participated in the ful software development lifecycle with requirements, solution design, development, QA implementation, and product support using Scrum and other Agile methodologies

Collaborate with team members and stakeholders in design and development of data environment

Preparing associated documentation for specifications, requirements, and testing.

Environment: AWS, GCP, Bigauery, Ges Bucket, G-Cloud Function, Apache Beam, Cloud Dataflow, Cloud Shell, Gsutil, Bq Command Line Utilities, Dataproc, Cloud Sa. Mysal, Posgres, Sql Server, Python, Scala, Spark, Hive, Spark -Sql

Client: Honey Well, IND Sep 2015 – July 2017

Role: Scala Developer

Responsibilities:

Imported required modules such as Keras and NumPy on Spark session, also created directories for data and output.

Read train and test data into the data directory as well as into Spark variables for easy access and proceeded to train the data based on a sample submission.

The images upon being displayed are represented as NumPy arrays, for easier data manipulation all the images are stored as NumPy arrays.

Created a validation set using Keras2DML in order to test whether the trained model was working as intended or not.

Defined multiple helper functions that are used while running the neural network in session. Also defined placeholders and number of neurons in each layer.

Created neural networks computational graph after defining weights and biases.

Created a TensorFlow session which is used to run the neural network as well as validate the accuracy of the model on the validation set.

After executing the program and achieving acceptable validation accuracy a submission was created thats stored in the submission directory.

Executed multiple SparkSQL queries after forming the Database to gather specific data corresponding to an image.

Environment: Scala, Python, PySpark, Spark, Spark ML Lib, Spark SQL, TensorFlow, NumPy, Keras, Powers!

Contact this candidate