Resume

Sign in

Hadoop developer / Data engineer

Location:
Minneapolis, MN
Salary:
65
Posted:
February 28, 2020

Contact this candidate

Resume:

Koushik Vinala

adb1t7@r.postjobfree.com

732-***-****

Seven years of professional experience in IT, which includes Analysis, Design, Coding, Testing, Implementation and support in Java and Big Data Technologies working with Apache Hadoop Eco-components.

Over three years of exclusive experience in Hadoop and its components like HDFS, Map Reduce, Apache Pig, Hive, Sqoop, Kafka, Spark, datastream, HBase and Oozie

Involved in writing the Pig scripts and Pig UDFs to pre-process the data for analysis

Experience in creating Hive External and Managed tables and writing queries on them

Hands on Experience in troubleshooting operational issues and identifying root causes of Hadoop Cluster

Expertise in managing data from multiple sources and transform large sets of data

Extensively used Sqoop to import data into HDFS from RDBMS and vice-versa.

Designing and creating HIVE external tables using shared meta-store instead of the derby with partitioning, dynamic partitioning and buckets.

Experience in Oozie and workflow scheduler to manage Hadoop jobs by Direct Acyclic Graph (DAG) of actions with control flows.

Experience in integrating Hive and HBase for effective operations.

Good understanding in working with various compression techniques like Avro, Snappy, LZO

Experienced in working with Spark ecosystem using Spark-SQL and Scala queries on different data file formats like .txt, .csv etc.

Hands on experience in migrating Map Reduce jobs into Spark RDD transformations using SCALA.

Good experience in Cloudera, Hortonworks &Apache Hadoop distributions.

Strong understanding of NoSQL databases and hands on work experience in writing applications on NoSQL databases like HBase, Cassandra and MongoDB, Redis, Neo4j.

Working knowledge on major Hadoop ecosystems PIG, HIVE, Sqoop, and Flume.

Experience in implementing Custom Partitions and Combiners for effective data distributions.

Experience in Writing Map Reduce jobs for text mining for predictive analysis

Experience in analyzing data using Cassandra QL, Hive QL and Pig Latin programs.

Experience in implementing Custom Partitions and Combiners for effective data distributions.

Good working knowledge with Map Reduce and Apache Pig

Have significant experience in application development using Java, J2EE, EJB, Hibernate, JDBC, Jakarta Struts, JSP and Servlets.

Experience in using various IDEs Eclipse, My Eclipse and repositories SVN and CVS.

Experience of using build tools Ant and Maven.

Working with relative ease with different working strategies like Agile, Waterfall and Scrum methodologies.

Excellent communication and analytical skills and flexible to adapt to evolving technology.

Languages

C, C++, Python, Java, J2EE, SQL, PL/SQL, Scala, UML, XML

Hadoop Ecosystem

HDFS, MapReduce, Spark Core, Spark Streaming, Spark SQL, Hive, Pig, Sqoop, Flume, Kafka, Oozie, Zookeeper.

Databases

Oracle 10g/11g, SQL Server, MYSQL, DB2

No SQL

HBase, Cassandra, MongoDB

Application / Web Servers

Apache Tomcat, JBoss, Mongrel, Web Logic, Web Sphere

Web Services

SOAP, REST

Operating systems

Windows, Unix, Linux

Microsoft Products

MS office, MS Visio, MS Project

Frameworks

Spring, Hibernate, Struts

Target Corporation, MN JAN 2019 to Till date

Hadoop Developer/Data Engineer

•Develop and test code to deliver functionality that meets the overall business strategy and objectives

•Build the work needed to implement features from the product backlog

•Size user stories

•Unit test code, create automated test scripts, and execute all other testing-related tasks

•Test deliverables against a user story’s acceptance tests

•Work closely with other team to ensure that features meet business needs

•Provide production support to all modules owned by team as per devops model.

•Extract files from external vendors and process the data.

•Build oozie workflows and co-ordinators and scheduling.

• Build Automation portal jobs in Datastream to call the oozie workflows.

•Create sharepoint locations for team and schedule internal data transfers and automate for ETL processing.

•Create new tables in Hive includes Hive External as well as Managed table.

•Create kafka topics and request postman tool to issue messages.

•Schedule jobs through oozie coordinators or Automation portal jobs to execute the kafka jobs.

•Requirements gathering, architecting, high level designing and modeling oracle application environments.

•Prepare the shift roaster for support activities.

•Sprint planning and iteration preparation to discuss and raise Jira stories based on the user story points.

•Defining the project timelines, schedules & costs for the subsequent phases.

•Developing and testing the solution for different phases as per the plan.

•Supporting the new environments for stabilization.

•Testing the entire To-Be system design for all the functional and business requirements.

•Testing the data migration as well as the integration of new billing system with the other applications.

•Complete the documentation of all changes implemented & new functionality added.

•Coordinate with stakeholders to take their inputs for the proposed solution and implement the same.

Tech Environment:

Hadoop, HDFS, Map Reduce, Apache Pig, Hive, Oozie, datastream, Kafka, SQOOP, Linux, MySQL, Spark, Hbase, HDP 2.6.5

T-Mobile, Atlanta, GA Jan 2017 to Dec 24 2018

Hadoop Developer/Spark Developer

Roles &Responsibilities:

Moved all crawl data flat files generated from various retailers to HDFS for further processing.

Written the Apache PIG scripts to process the HDFS data.

Created Hive tables to store the processed results in a tabular format.

Developed the sqoop scripts in order to make the interaction between Pig and MySQL Database.

Writing the script files for processing data and loading to HDFS

Writing CLI commands using HDFS.

Developed the UNIX shell scripts for creating the reports from Hive data.

Completely involved in the requirement analysis phase.

Experience in architecture and creating data lake from different source systems like RDBMS and Teradata.

Created two different users (hduser for performing hdfs operations and mapred user for performing map reduce operations only)

Ensured NFS is configured for Name Node

Setting Password less Hadoop

Written PIG scripts to process the Credit Card and Debit Card Transactions for Active customers by joining the data from HDFS and Hive using HCatalog for various merchants

Worked on HBase for generating additional columns on the data lake zones

Created Hive scripts for joining the raw data with the lookup data and for some aggregative operations as per the business requirement.

Creating external and internal tables on hadoop data lakes to manage up-steam and down-stream data flow.

Setting up cron job to delete Hadoop logs/local old job files/cluster tempfiles

Setup Hive with MySQL as a Remote Metastore

Worked extensively on creating sqoop jobs to manage data flow into data lake

Moved all log/text files generated by various products into HDFS location

Written Map Reduce code that will take input as log files and parse the logs and structure them in tabular format to facilitate effective querying on thelog data

Loading data from UNIX file system to HDFS and vice versa.

Implemented Data Ingestion in real time processing using Kafka.

Created External Hive Tables on top of parsed data.

Environment: Hadoop, HDFS, Map Reduce, Apache Pig, Hive, SQOOP, Linux, Data Lake, MySQL, Spark, Hbase, Hortonworks, HDP 2.6.5

Vodafone, Pune, India. Jan 2016 to July 2016

Java Developer

Roles &Responsibilities:

Analyzing the feasibility Documents

Coding the business logic methods in core java.

Involved in development of the Action classes and Action Forms based on the Struts framework.

Participated in client-side validation and server-side validation.

Involved in creation of struts configuration file and validation file for skip module using struts framework.

Developed java programs, JSP pages and servlets using Spring framework.

Involved in creating database tables, writing complex TSQL queries and stored procedures in the SQL server.

Worked with AJAX framework to get the asynchronous response for the user request and used JavaScript for the validation.

Used EJBs in the application and developed Session beans to implement business logic at the middle tier level.

Actively involved in writing SQL using SQL Query Builder.

Used JAXB to read and manipulate the xml properties.

Used JNI for calling the libraries and other implemented functions in C language.

Handling Server Related issues, new requirement handling, changes and patch movements.

Developed the Restful Web Services for various XSD schemas.

Used Servlets to implement Business components.

Designed and Developed required Manager Classes for database operations.

Developed various Servlets for monitoring the application.

Designed the UML class diagram, Sequence diagrams for Trade Services.

Designed the complete Hibernate mapping for SQL Server for PDM.

Designed the complete JAXB classes mapping for various XSD schemas.

Developed the Restful Web Services for various XSD schemas.

Involved in writing JUnit test Classes for performing Unit testing.

Environment: Eclipse neon, jdk1.8.0, Java, J2EE, Servlets, JSP, EJB, xml, SQL server, Struts, JUnit and Eclipse, SQL, UNIX, UML, Apache-Maven3.0.3

Birla Soft, Hyderabad, India Jun 2014 to Dec 2015

Java Developer

Roles &Responsibilities:

Identifying reviewing, assessing and resolving production issues

Configure and maintain the associated application components and environments (as required).

Provide application support to management, team members and end users

Having experience on the sales functionality and order management

Worked on the Email template creation with HTML code as per the requirements

Email notification configured based on the requirement

Involved in writing programs for XA transaction management on multiple databases of the application.

Developed java programs, JSP pages and servlets using Cantata Struts framework.

Involved in creating database tables, writing complex TSQL queries and stored procedures in the SQL server.

Worked with AJAX framework to get the asynchronous response for the user request and used JavaScript for the validation.

Used EJBs in the application and developed Session beans to implement business logic at the middle tier level.

Actively involved in writing SQL using SQL Query Builder.

Involved in coordinating the on-shore/Off-shore development and mentoring the new team members.

Extensively Used Ant tool to build and configure J2EE applications and used Log4J for logging in the application

Used JAXB to read and manipulate the xml properties.

Used JNI for calling the libraries and other implemented functions in C language.

Used prototype Moo Tools and script.aculo.us for fluid User Interface.

Involved in fixing defects and unit testing with test cases using JUnit.

Involved in Configured business components, Views, applets, Controls, Menus and other objects to meet the business Requirements

Environment: jdk1.8.0, Java, Servlets, JSP, xml, SQL server, JUnit and Eclipse, Unix, UML, Apache-Maven3.0.3, EJB, Servlets, XSLT, CVS, J2EE, AJAX, Struts, Hibernate, ANT, Tomcat, JMS, Log4J, Oracle 10g, Eclipse, Solaris, JUnit and Windows 7/XP

Master’s in Computer Science and Information Systems

PROFESSIONAL SUMMARY

TECHNOLOGIES

PROFESSIONAL EXPERIENCE

EDUCATION



Contact this candidate