Hadoop Developer
Resume:
Avinash
**********@*****.***
Professional Summary:
Around 7 years of experience in IT and 4+ years hands on experience in Hadoop ecosystem’s implementation, maintenance, ETL and Big Data analysis operations.
Excellent knowledge/understanding of Hadoop architecture and its components.
Hands-on experience in installing, configuration and using Hadoop ecosystem components such as HDFS, MapReduce, PIG, Hive, Sqoop, Spark, Kafka, and Flume.
Full-scale knowledge of Hadoop components such as HDFS, Job Tracker, Name Node, Data Node.
Good experience in Map Reduce Programming, Pig Scripting and analyzing data using HiveQL, Pig Latin, and HBase.
Developed enterprise applications using Scala.
Experience in designing and developing POCs in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle. Experience in managing and monitoring Hadoop cluster using Cloudera Manager.
Experience in writing custom UDF’s which extends Hive and Pig core functionalities.
Experience in working with different NOSQL databases like HBase.
Handled different file formats like Parquet, Avro files, RC files using different SerDes in Hive.
Experience in importing and exporting data using Sqoop from HDFS/Hive/HBase to Relational Database Systems.
Experience creating real-time data streaming solutions using Apache Spark core, Spark SQL, Kafka, Spark Streaming and Apache Storm.
Converted Map Reduce Applications to Spark.
Performed Data Ingestion from multiple disparate sources and systems using Kafka.
Transformed big data requirements into Hadoop-driven technologies working along with BI team.
Experience in job workflow schedulers and monitoring applications such as Oozie and Zookeeper.
Experience in dumping shared data into Hdfs from MySQL by writing shell scripts.
Good knowledge in the collection of log files from different sources using Flume and Kafka.
Experience in Core Java, Java Virtual Machines and multi-thread processing.
Proficiency in working with databases like Oracle, MySQL.
Extensive experience in writing stored procedures and functions using SQL and PL/SQL.
Good Knowledge on developing Java Applications using IDEs like Eclipse, NetBeans.
Technical Skills
Hadoop/Big Data:
HDFS, MapReduce, HBase, Pig, Hive, Sqoop, Spark, Zookeeper, Impala, Oozie, Cassandra, MongoDB.
Methodologies:
Agile, UML, Design Patterns (Core Java and J2EE)
Programming:
Java, J2EE, XML, Scala, Unix Shell scripting, HTML.
Java/J2EE Technologies
JDBC, JNDI, JSON, JSTL, RMI, JMS, Java Script, JSP, Servlets, EJB, JSF, jQuery, AngularJS.
Modelling Tools
Rational Rose, Star UML, Visual paradigm for UML
Databases:
Oracle 11g/10g, MS-SQL Server, MySQL, MS-Access
Operating Systems:
Windows 7/8/10, UNIX, Linux, Ubuntu.
Professional Experience:
Blazent – Livonia, Michigan January 2016 - Present
Role: Hadoop Developer
Blazent Analysis project has been implemented using Hadoop and Big Data technologies. Blazent has recognized the value of being able to gather and analyze information coming from customer interactions in the call center such as call center emails, online chat, Twitter, Facebook etc. This enables the company to improve service quality, increase customer loyalty and ultimately drive revenue.
Responsibilities:
Analyzing the requirement to setup a cluster.
Installed and configured Hadoop, MapReduce, HDFS (Hadoop Distributed File System), developed multiple MapReduce jobs in Java.
Developed Map Reduce programs in Java for parsing the raw data and populating Staging Tables.
Developed Spark scripts by using Scala shell commands as per the requirement.
Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
Developed Scala scripts, UDFFs using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
Importing and exporting data into HDFS and Hive using Sqoop.
Experienced in analyzing data with Hive and Pig.
Writing Pig scripts to process the data.
Developed PIG Latin scripts to extract the data from the web server output files to load into HDFS.
Integrating bulk data into Cassandra file system using MapReduce programs.
Involved in HBase setup and storing data into HBase, which will be used for further analysis.
Installed and configured Hive and written Hive UDFs.
Involved in creating Hive tables, loading with data and writing Hive queries using the HiveQL which will run internally in the map-reduce way.
Extracted the data from MySQL into HDFS using Sqoop.
Developed Scala and SQL code to extract data from various databases Champion new innovative ideas around the Data Science and Advanced Analytics Practices Creatively communicated and presented models to business customers and executives, utilizing a variety of formats and visualization methodologies.
Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.
Used HiveQL to analyze the partitioned and bucketed data and compute various metrics for reporting.
Environment: Java (JDK1.7), Java 7, Hadoop, MapReduce, HDFS, Hive, Sqoop, HBase, Pig, Oozie, Kerberos, Linux, Scala, Shell Scripting, Oracle 11g.
Foot Locker – Wisconsin June 2014 - January 2016
Role: Hadoop Developer
Foot Locker Evaluation project is a repository of all the markets details and their products available in different regions to model dealership performance. Using a statistical model, we need to determine the percentage of buyers who and where they buy luxury brands, those who prefer mid-sized and so on for each sales category based on the product availability. This analysis is to make better decisions, understand customers and provide better service.
Responsibilities:
Mainly worked on Hive queries to categorize data of different claims.
Integrated the Hive warehouse with HBase.
Involved in loading data from LINUX file system to HDFS. Written customized Hive UDFs in Python where the functionality is too complex.
Implemented Partitioning, Dynamic Partitions, Buckets in Hive.
Designing and creating Hive external tables using shared Meta-store instead of the derby with partitioning, dynamic partitioning, and buckets.
Responsible for managing the test data coming from different sources Reviewing peer table creation in Hive, data loading, and queries.
Gained experience in managing and reviewing Hadoop log files.
Deployed Hadoop Cluster in Fully Distributed and Pseudo-distributed modes.
Experience in managing and monitoring Hadoop cluster using Cloudera Manager.
Involved in scheduling Oozie workflow engine to run multiple Hive and Pig jobs involved unit testing, interface testing, system testing and user acceptance testing of the workflow tool.
Supported in setting up QA environment and updating configurations for implementing scripts with Pig, Hive, and Sqoop.
Environment: Hadoop, HDFS, Hive, Map Reduce, Core Java, Pig, Oracle, MySQL.
Capital IQ – Hyderabad, India May 2012 – June 2014
Role: Java/J2EE Developer
The product developed was Online Banking system. The Online banking system developed provides personal savings details of the clients. It offers a broad array of financial clients and as per the user's account status offers him to take up the new schemes enrolling with the bank. Upon the user's request, it generates the reports of his details and submits it to the user.
Responsibilities:
Responsible for understanding the business requirement.
Worked with Business Analyst and helped to represent the business domain details in technical specifications.
Also helped to develop UML Diagrams: Use Cases, Activity Diagram, Sequence Diagram, Class Diagram.
Was also actively involved in setting coding standards and writing related documentation.
Developed the Java Code using Eclipse as IDE.
Developed JSPs and Servlets to dynamically generate HTML and display the data to the client side.
Developed application on Struts MVC architecture utilizing Action Classes, Action Forms and validations.
Code Review & Debugging using Eclipse Debugger.
Involved in the design and decision making for Hibernate or Mapping. Developed Hibernate Mapping file (. hbm.xml) files for mapping declarations.
Configured Queues in WebLogic server where the messages, using JMS API, were published.
Consumed Web Services (WSDL, SOAP,) from the third party for authorizing payments to/from customers. Writing/Manipulating the database queries, stored procedures for Oracle9i.
Environment: Java, /J2EE, Oracle9i, JSP1.1, HTML, JavaScript, JMS, Servlets, UML, XML, Eclipse, Web Services, WSDL, SOAP, ANT, JUnit, Log4j.
Ritkam Technologies - Chennai, Tamil Nadu June 2010 - May 2012
Role: Java developer
The application is about connecting to one or more remote systems and transfers the messages and files between the remotely connected systems. The socket programming was used for connecting to remote systems. The project was developed using core and advanced java along the PL/SQL Database.
Responsibilities:
Involved in the process Design, Coding and Testing phases of the software development cycle.
Developed Servlets, JSP pages, Beans, JavaScript.
Involved in developing a module for the transformation of files across the remote systems using JSP and servlets.
JSP’s had HTML tags and JSP tags that were used as to develop the view of the module and servlets acted as backend java classes that were used to connect to Database retrieve files from there and send it across the systems or store the receiving files from other systems.
Development of Various Reports using Java and SQL queries
Participation in Design, Issue Resolution, and Sustainment meetings.
Environment: Java, Servlets, JSP, Oracle, Windows NT and UNIX, Tomcat
Education: Bachelor’s in Computer Science and Engineering, Jawaharlal Nehru Technological University – India
Contact this candidate