Resume

Sign in

Data Java Developer

Location:
Dallas, Texas, United States
Posted:
January 23, 2018

Contact this candidate

Resume:

PRAVEEN KUMAR

ac35ph@r.postjobfree.com 314-***-**** ext 751

Professional Summary

Hadoop Developer with 8+ years of overall IT experience in a variety of industries, which includes hands on experience in Big Data technologies.

Have 4+ years of comprehensive experience in Big Data processing using Hadoop and its ecosystem (MapReduce, Pig, Hive, Sqoop, Flume, Spark, Kafka and HBase).

Worked on installing, configuring, and administrating Hadoop cluster for distributions like Cloudera Distribution.

Efficient in writing MapReduce Programs and using Apache Hadoop API for analyzing the structured and unstructured data.

Expert in working with Hive data warehouse tool-creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the HiveQL queries.

Debugging Pig and Hive scripts and optimizing MapReduce job and debugging Map reduce job.

Administrator for Pig, Hive and Hbase installing updates patches and upgrades.

Hands-on experience in managing and reviewing Hadoop logs.

Good knowledge about YARN configuration.

Expertise in writing Hadoop Jobs for analyzing data using Hive QL (Queries), Pig Latin (Data flow language), and custom MapReduce programs in Java.

Extending Hive and Pig core functionality by writing custom UDFs.

Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.

Hands on experience in configuring and working with Flume to load the data from multiple sources directly into HDFS.

Good working knowledge on NoSQL databases such as Hbase, MongoDB and Cassandra.

Used Hbase in accordance with PIG/Hive as and when required for real time low latency queries.

Knowledge of job workflow scheduling and monitoring tools like Oozie (hive, pig) and Zookeeper (Hbase).

Good working experience on Spark (spark streaming, spark SQL), Scala and Kafka.

Worked on reading multiple data formats on HDFS using Scala.

Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.

Good experience in creating and designing data ingest pipelines using technologies such as Apache Storm- Kafka.

Integrated Apache Storm with Kafka to perform web analytics. Uploaded click stream data from Kafka to Hdfs, Hbase and Hive by integrating with Storm

Developed various shell scripts and python scripts to address various production issues.

Developed and designed automation framework using Python and Shell scripting

Generated Java APIs for retrieval and analysis on No-SQL database such as HBase and Cassandra.

Experience in AWS EC2, configuring the servers for Auto scaling and Elastic load balancing.

Configuring AWS EC2 instances in VPC network & managing security through IAM and Monitoring servers health through Cloud Watch.

Good Knowledge of data compression formats like Snappy, Avro.

Hands on experience in developing the applications with Java, J2EE, J2EE - Servlets, JSP, EJB, SOAP, Web Services, JNDI, JMS, JDBC2, Hibernate, Struts, Spring, XML, HTML, XSD, XSLT, PL/SQL, Oracle10g and MS-SQL Server RDBMS.

Dealt with huge transaction volumes while interfacing the front end application written in Java, JSP, Struts, Hibernate, SOAP Web service and with Tomcat Web server.

Delivered zero defect code for three large projects which involved changes to both front end (Core Java, Presentation services) and back-end (Oracle).

Experience with all stages of the SDLC and Agile Development model right from the requirement gathering to Deployment and production support.

Involved in daily SCRUM meetings to discuss the development/progress and was active in making scrum meetings more productive.

Also have experience in understanding of existing systems, maintenance and production support, on technologies such as Java, J2EE and various databases (Oracle, SQL Server).

Technical Skills

Big Data

Cloudera Distribution, HDFS, Zookeeper, Yarn, Data Node, Name Node, Resource Manager, Node Manager, Mapreduce, PIG, SQOOP, Hbase, Hive, Flume, Cassandra, MongoDB, Oozie, Kafka, Spark, Storm, Scala, Impala

Operating System

Windows, Linux, Unix.

Languages

Java, J2EE, SQL, PYTHON, Scala

Databases

IBM DB2, Oracle, SQL Server, MySQL, PostGres

Web Technologies

JSP, Servlets, HTML, CSS, JDBC, SOAP, XSLT.

Version Tools

GIT, SVN, CVS

IDE

IBM RAD, Eclipse, IntelliJ

Tools

TOAD, SQL Developer, ANT, Log4J

Web Services

WSDL, SOAP.

ETL

Talend ETL, Talend Studio

Web/App Server

UNIX server, Apache Tomcat

Professional Experience

CapitalOne, Dallas, TX Jan 2017 – Till Date

Sr. Data Engineer

Responsibilities:

Handled the importing of data from various data sources like Oracle and MySQL using SQOOP performed transformation using Hive and loaded the data into HDFS.

Participate in requirement gathering and documenting the business requirements by conducting grooming sessions/meetings with various business users.

Involved in creating Hive tables and written multiple Hive queries to load the hive tables for analyzing the market data coming from distinct sources.

Experience in NoSQL databases such as HBase and Cassandra.

Created extensive SQL queries for data extraction to test the data against the various databases.

Responsible for importing data to HDFS using Sqoop from different RDBMS servers and exporting data using Sqoop to the RDBMS servers after aggregations for other ETL operations.

Created Partitioning, Bucketing, Map side Join, Parallel execution for optimizing the hive queries.

Generated Java APIs for retrieval and analysis on No-SQL database such as HBase and Cassandra.

Developed Simple to complex Map Reduce Jobs using Hive and Hbase.

Used Impala to read, write and query the data in HDFS.

Orchestrated various Sqoop queries, Pig scripts, Hive queries using Oozie workflows and sub-workflows.

Responsible for handling different data formats like Avro, Parquet and ORC formats.

Involved in generating analytics data using Map/Reduce programs written in Python.

Used Kafka to load data in to HDFS and move data into NoSQL databases.

Involved in creation and designing of data ingest pipelines using technologies such as Apache Kafka.

Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications.

Experience in custom aggregate functions using Spark SQL and performed interactive querying.

Implemented Apache Spark data processing project to handle data from RDBMS and streaming sources.

Worked on Spark SQL and Data frames for faster execution of Hive queries using Spark SqlContext.

Performed analysis on implementing Spark using Scala.

Designed batch processing jobs using Apache Spark to increase speeds by ten-fold compared to that of MR jobs.

Experience in AWS EC2, configuring the servers for Auto scaling and Elastic load balancing.

Configuring AWS EC2 instances in VPC network & managing security through IAM and Monitoring servers health through Cloud Watch.

Designed the ETL process and created the high level design document including the logical data flows, source data extraction process, the database staging and the extract creation, source archival, job scheduling and Error Handling.

Worked on Talend ETL tool and used features like context variable and database components like input to oracle, output to oracle, tFile compare, tFile copy, to oracle close ETL components.

Created ETL Mapping with Talend Integration Suite to pull data from Source, apply transformations, and load data into target database.

Active member for developing POC on streaming data using Apache Kafka and Spark Streaming.

Involved in daily SCRUM meetings to discuss the development/progress and was active in making scrum meetings more productive.

Environment: Hadoop, Java, AWS, MapReduce, HDFS, Hive, Pig, Sqoop, Flume, Python, Spark, Impala, Scala, Kafka, Shell Scripting, Eclipse, Cloudera, MySQL, Talend, Cassandra

Alcon-Novartis Pharma, Fort Worth, Texas Sep 2015 – Dec 2016

Data Engineer

Responsibilities:

Involved with ingesting data received from various relational database providers, on HDFS for analysis and other big data operations.

Creating Hive tables to import large data sets from various relational databases using Sqoop and export the analyzed data back for visualization and report generation by the BI team.

Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time.

Configured deployed and maintained multi-node Dev and Test Kafka Clusters.

Developed Spark scripts by using Java, and Python shell commands as per the requirement.

Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.

Developed Scala scripts, UDFs using both Data frames/SQL/Data sets and RDD/MapReduce in Spark 1.6 for Data Aggregation, queries and writing data back into OLTP system through Sqoop.

Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.

Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.

Developed Pig Scripts, Pig UDFs and Hive Scripts, Hive UDFs to analyze HDFS data.

Extracted files from MongoDB through Sqoop and placed in HDFS and processed.

Maintained the cluster securely using Kerberos and making the cluster up and running all the times.

Implemented optimization and performance testing and tuning of Hive and Pig.

Experience in migrating HiveQL into Impala to minimize query response time.

Developed a data pipeline using Kafka to store data into HDFS.

Worked on reading multiple data formats on HDFS using Scala

Worked on Spark SQL and Data frames for faster execution of Hive queries using Spark SqlContext.

Performed analysis on implementing Spark using Scala.

Implemented spark sample programs in python using pyspark.

Responsible for creating, modifying topics (Kafka Queues) as and when required with varying configurations involving replication factors and partitions.

Written shell scripts and Python scripts for automation of job.

Environment: Cloudera, HDFS, Hive, HQL scripts, Map Reduce, Java, HBase, Pig, Sqoop, Kafka, Impala, Shell Scripts,Python Scripts, Spark, Scala, Oozie.

Arrow Electronics, Englewood, CO Jun 2014 – Aug 2015

Hadoop Developer

Responsibilities:

Developed MapReduce programs to parse the raw data, and create intermediate data which would be further used to be loaded into Hive portioned data.

Involved in creating Hive ORC tables, loading the data into it and writing Hive queries to analyze the data.

Involved in data ingestion into HDFS using Sqoop for full load and Flume for incremental load on variety of sources like web server, RDBMS and Data API’s.

Performed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing

Used different file formats like Text files, Sequence Files, Avro, Optimized Row Columnar (ORC)

Ingest real-time and near-real time (NRT) streaming data into HDFS using Flume

Expertise in creating TWS Jobs and Jobstreams and automate them as per schedule

Worked on Golden Gate replication tool to get data from various data sources into HDFS

Worked on HBase for support enterprise production and loading data into HBASE using SQOOP.

Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.

Exported the data from Avro files and indexed the documents in ORC file format.

Responsible for created Technical Specification documents for the generated extracts

Involved in performance tuning using Partitioning, bucketing of Hive tables

Created UDFs to calculate the pending payment for the given customer data based on last day of every month and used in Hive Scripts.

Involved in writing shell scripts to run the jobs in parallel and increase the performance

Involved in running TWS jobs for processing millions of records using ITG.

Environment: Hadoop, HDFS, MapReduce, Yarn, Hive, PIG, Oozie, Sqoop, HBase, Flume, Linux, Shell scripting, Java, Eclipse, SQL

CITI Bank, Dallas, TX Jan 2013 – May 2014

Hadoop Developer

Responsibilities:

Developed Pig Scripts, Pig UDFs and Hive Scripts, Hive UDFs to analyze HDFS data.

Used Sqoop to export data from HDFS to RDBMS.

Having experience on Hadoop eco system components HDFS, MapReduce, Hive, Pig, Sqoop and HBase.

Expertise with web based GUI architecture and development using HTML, CSS, AJAX, JQuery, Angular Js, and JavaScript.

Developed Map Reduce programs for some refined queries on big data.

Involved in loading data from UNIX file system to HDFS.

Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.

Extracted the data from Databases into HDFS using Sqoop.

Handled importing of data from various data sources, performed transformations using Hive, PIG and loaded data into HDFS.

Used PIG predefined functions to convert the fixed width file to delimited file

Used HIVE join queries to join multiple tables of a source system and load them into Elastic Search Tables.

Automated the workflow using shell scripts.

Manage and review Hadoop log files. Implemented lambda architecture as s solution to a problem.

Involved in analysis, design, testing phases and responsible for documenting technical specifications.

Very good understanding of Partitions, bucketing concepts Managed and External tables in Hive to optimize performance.

Expertise in writing Hadoop Jobs for analyzing data using Hive QL (Queries), Pig Latin (Data flow language), and custom MapReduce programs in Java.

Experienced in running Hadoop streaming jobs to process terabytes data in Hive and designed both.

Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.

Created reports for the BI team using Sqoop to export data into HDFS and Hive.

Environment: Hadoop, HDFS, MapReduce, Yarn, Hive, PIG, Oozie, Sqoop, HBase, Flume, Linux, Shell scripting, Java, Eclipse, SQL

ADP-India, Hyderabad, India Jan 2011 – Dec 2012

Java Developer

Responsibilities:

Also helped developing UML Diagrams: Use cases, Activity diagram, Sequence diagram, class Diagram.

Developed JSPs and Servlets to dynamically generate HTML and display the data to the client side.

Developed application on Struts MVC architecture utilizing Action Classes, Action Forms and validations

Was responsible in implementing various J2EE Design Patterns like Service Locator, Business Delegate, Session Facade and Factory Pattern.

Involved in the design and decision making for Hibernate OR Mapping.

Developed Hibernate Mapping file (.hbm.xml) files for mapping declarations.

Configured Queues in Web Logic server where the messages, using JMS API, were published.

Consumed Web Services (WSDL, SOAP, and UDDI) from third party for authorizing payments to/from customers.

Writing/Manipulating the database queries, stored procedures for Oracle9i.

Developed the services by following a full flown Test Driven Development.

Interacting with the system analysts, business users for design & requirement clarifications.

Designed front end pages using Jsp, HTML, Angular JS, JQuery, JavaScript, Css and Ajax calls to get the required data from backend.

Designed and developed this application using Spring MVC.

Spring Framework IOC (Inversion of Control) design pattern is used to have relationships between application components.

Developed the DAO layer for the application using Hibernate and JDBC.

Implemented Restful web services with JAX-RS (Jersey).

Used JMS (Java Message Service) to send, receive and read messages in the application.

Used Junit for testing.

Used databases like Oracle 10g, DB2 and wrote complex SQL statements, PL/SQL Procedures, Cursors to retrieve data from DB.

Used extensively Eclipse and RAD in development and debugging the application.

Used maven as a project build, dependency and management tool.

Environment: Java,/J2EE, Eclipse, Web Logic Application Server, Oracle, JSP1, HTML, JavaScript, JMS, Servlets, UML, XML, Struts, Web Services, WSDL, SOAP, UDDI, ANT, JUnit, Log4j.

COE Secure Pvt Ltd - Hyderabad, India May 2009 - Dec 2010

Java Developer

Responsibilities:

Responsible for understanding the business requirement.

Worked with Business Analyst and helped representing the business domain details in technical specifications.

Also helped developing UML Diagrams: Use cases, Activity diagram, Sequence diagram, class Diagram.

Was also actively involved in setting coding standards and writing related documentation.

Developed the Java Code using Eclipse as IDE.

Developed JSPs and Servlets to dynamically generate HTML and display the data to the client side.

Developed application on Struts MVC architecture utilizing Action Classes, Action Forms and validations.

Tiles were used as an implementation of Composite View pattern

Was responsible in implementing various J2EE Design Patterns like Service Locator, Business Delegate, Session Facade and Factory Pattern.

Code Review & Debugging using Eclipse Debugger.

Involved in the design and decision making for Hibernate OR Mapping.

Developed Hibernate Mapping file (.hbm.xml) files for mapping declarations.

Configured Queues in WebLogic server where the messages, using JMS API, were published.

Consumed Web Services (WSDL, SOAP, and UDDI) from third party for authorizing payments to/from customers.

Writing/Manipulating the database queries, stored procedures for Oracle9i.

Environment: Java,/J2EE, Eclipse, Web Logic Application Server, Oracle, JSP, HTML, JavaScript, JMS, Servlets, UML, XML, Struts, WSDL, SOAP, UDDI, ANT, JUnit, Log4j.



Contact this candidate