Data Hadoop

Location:

Austin, TX

Posted:

March 22, 2021

Contact this candidate

Resume:

Ramesh Nunna

Email:*********@*****.***

Mobile: 630-***-****

Professional Summary

Having 11+ years of IT experience including 5+ years of experience with Hadoop Ecosystem in installation

and configuration of different Hadoop eco-system components in the existing cluster and 6 years of experience on IBM Tivoli storage manager administration on Linux/AIX/Wndows platforms.

Experience in deploying and managing the multi-node development, testing and production Hadoop cluster with different Hadoop components (HDFS,YARN,HIVE, HBASE[Phoenix &Splice], SOLR, PIG, Spark,KAFKA, SQOOP, OOZIE, RANGER, ZOOKEEPER) using Apache Ambari.

Experience on Horton works and Cloudera manager.

Strong knowledge on Hadoop HDFS architecture and Map-Reduce framework.

Experience in improving the Hadoop cluster performance by considering the OS kernel, Storage, Networking, Hadoop HDFS and Map-Reduce by setting appropriate configuration parameters.

Experience in administering the Linux systems to deploy Hadoop cluster and monitoring the cluster using Ambari.

Experience in upgrading Hadoop cluster from current version to minor version upgrade as well as to major versions.

Experience in PODIUM to create connections from up streams to Hadoop cluster.

Experience in using Zookeeper for coordinating the distributed applications.

Experience in managing Hadoop infrastructure like commissioning, decommissioning, log rotation, rack topology implementation.

Experience in managing the cluster resources by implementing fair and capacity scheduler.

Experience in benchmarking, performing backup and disaster recovery of Name Node metadata and important sensitive data residing on cluster.

Strong knowledge in configuring Name Node High Availability.

Experience in handling multiple relational databases: MySQL, SQL Server.

Assisted Developers with problem resolution.

Worked on GCP and AWS.

Ability to play a key role in the team and communicates across the team.

Global Service Delivery experience by bringing together resources to accomplish organizational goals using ITIL framework.

Effective problem-solving skills and outstanding interpersonal skills. Ability to work independently as well as within a team environment. Driven to meet deadlines. Ability to learn and use new technologies quickly.

Worked on setting up Name Node high availability for major production cluster and designed Automatic failover control using zookeeper and quorum journal nodes.

Setting up automated 24x7 monitoring and escalation infrastructure for Hadoop cluster using Ambari

Experienced in Linux Administration and TSM Administration.

Educational Qualifications

Master of computer Applications from Kakatiya University, India in 2008.

Career Objective

To be part of an organization with an objective to accept the challenges and to work towards achieving the goals of organization that will build on my skills and provide me with ample scope for growth.

Technical Skills

Hadoop Ecosystems

HDFS, YARN,Hive, Sqoop, Spark, Kafka,Splunk,SOLR Zookeeper, HBase with Phoenix and &splice,Oozie,

Kerberos, Ranger.

Operating System

Windows, Linux, AIX,Ubuntu, AWS,

RDBMS

MYSQL, Oracle, DB2, MSSQL SERVER,TERADATA.

Languages

C, C++, Java, shell scripting, Python, SQL,scala.

Other Tools/tech

PODIUM,SVN,1Automation,Service Now, Win SCP, kubernetes.

Professional Experience

Hadoop Administrator

Only It Consulting LLC.

VISA Inc- Austin,Texas Mar 2019 to Till Date.

Job Responsibilities

Providing production support for large scale Hadoop clusters.

Working on Hive,spark,Hbase as a service.

Worked on Hbase performance tuning.

Worked on Maintenance activities by stopping and starting Hadoop clusters.

Performed patching by using VVMS automation tool.

Worked on Spark and HIVE job failures.

Configured alert setup for newly onboarded clusters.

Worked on Kafka with Apache and confluent in production environment.

Implemented performance tuning on AMS in Large scale clusters.

Worked on LINUX servers for basic troubleshooting on Hadoop nodes.

Worked on Deep dive issue analysis specially on Kafka and Hbase areas.

Experienced in monitoring kafka cluster by using Kaas portal.

Providing production support for Kafka Clusters.

Performing OS and security patching on production nodes through automation tool.

Co-ordinating with linux team about Hardware failures on production nodes.

Participating in code deployments and coordinating with developers.

Worked on Hadoop cluster maintenance activities End to End.

Working on Creating HIVE Tables and VIEWS on Hadoop Cluster.

Troubleshooting issues with jobs and coordinating dev team to fix them.

Worked on ranger policy creation and updations.

Having knowledge on setting up security like Kerberos, Ranger .

Performed pre-implementation tasks on the CentOS operating system to accommodate installation Hadoop distribution.

Planning of nodes for setting up various daemons.

Configured Zookeeper to implement node coordination, in clustering support.

Worked on YARN Containers with performance tuning.

Worked on security patching and java version upgrades.

Monitoring Hadoop cluster from AMBARI.

Commission of Hadoop New Nodes.

Monitoring Hadoop Logs for Job Failures.

Worked on Production clusters with multi-tenant.

Worked on Hive performance tuning.

Involved in testing HDFS, Hive, Pig and Map Reduce access for the new users.

Monitoring Hadoop Ambari cluster and jobs.

Troubleshooting failed jobs, Performance Tuning, Automation using shell script.

Performed various Ambari and HDP upgrades.

Worked on different Hadoop eco system tools like HDFS, YARN, Hive, Spark and HBase.

Improved HBase performance and Yarn performance by fine tuning parameters.

Extremely good knowledge and experience with Hbase,Kafka, Map Reduce, Spark Streaming, SparkSQL for data processing.

Import data from external table into HIVE by using scripts.

Configured various property files like core-site.xml, hdfs-site.xml, mapred-site.xml based upon the job requirement.

Created table in hive and use static, dynamic partition for data slicing mechanism.

Working experience with monitoring cluster, identifying risks, establishing good practices to be followed in shared environment

Good understanding on cluster configurations and resource management using YARN.

Worked on tuning the performance of MapReduce Jobs.

Responsible to manage data coming from different sources.

Load and transform large sets of structured, semi structured and unstructured data.

Experience in managing and reviewing Hadoop log files.

Performed Namenode backups,JN backups and MYSQL DB backups.

Job management using Fair scheduler.

Worked on Kubernetes containers.

Hadoop Administrator

Only It Consulting LLC.

IHG- Atlanta,Georgia. Sep 2018 to Mar 2019

Job Responsibilities

Responsible for architecting Hadoop clusters Translation of functional and technical requirements into detailed architecture and design.

Installed and configured multi-nodes fully distributed Hadoop cluster of large number of nodes.

Addressing and Troubleshooting issues daily basis.

File system management and monitoring.

Supported Kafka clusters in production environment.

Provided Hadoop, OS, Hardware optimizations.

Worked on HDP 3.1.5 Installation in Lower Environments and Production.

Worked on Kerbroes,Ranger.

Worked on Security patching activities.

Worked on AWS & GCP cloud.

Worked with popular Hadoop distribution like Hortonworks.

Performed Hadoop name node metadata backups during Maintenance activities.

Worked independently with Hortonworks support for any issue/concerns with Hadoop cluster.

Implementing Hadoop Security on Hortonworks Cluster.

Installed and configured Hadoop ecosystem components like Map Reduce, Hive, Pig, Sqoop, HBase, Zookeeper and Oozie.

Worked on Production Environment with different applications.

Worked on YARN containers over Hadoop Cluster in GCP.

Good experience in troubleshoot production level issues in the cluster and its functionality.

Backed up data on regular basis to a remote cluster using DistCp.

Regular Commissioning and Decommissioning of nodes depending upon the amount of data.

Maintaining Cluster in order to remain healthy and in optimal working condition.

Handle the upgrades and Patch updates.

Hadoop Administrator

Only It Consulting LLC.

Charter communications- Charlotte,NC Feb 2018 to Sep 2018

Job Responsibilities

Responsible for architecting Hadoop clusters Translation of functional and technical requirements into detailed architecture and design.

Installed and configured multi-nodes fully distributed Hadoop cluster of large number of nodes.

File system management and monitoring.

Worked on GCP HDP clusters.

Performed copy of date over GCP from one cluster to another cluster.

Installed and configured Hadoop ecosystem components like Map Reduce, Hive, Pig, Sqoop, HBase, Zookeeper and Oozie.

Worked on Production clusters.

Setup alerts by using Python and shell scripting.

Involved in testing HDFS, Hive, Pig and Map Reduce access for the new users.

Cluster maintenance as well as creation and removal of nodes using Apache Ambari

Worked on setting up high availability for major production cluster and designed automatic failover control using zookeeper and quorum journal nodes.

Implemented capacity scheduler to allocate fair amount of resources to small jobs.

Performed operating system installation, Hadoop version updates using automation tools.

Configured Oozie for workflow automation and coordination.

Implemented rack aware topology on the Hadoop cluster.

Importing and exporting structured data from different relational databases into HDFS and Hive using Sqoop.

Configured Zookeeper to implement node coordination, in clustering support.

Rebalancing the Hadoop Cluster.

Installed and configured Hadoop security tools Knox, Ranger and enabled Kerberos

Managing cluster performance issues.

Environment: Hortonworks (HDP 2.5), Ambari 2.4, HDFS, Java, Shell Scripting, Splunk, Python,Hive, Spark, Sqoop, Linux, SQL, Cloudera, Zookeeper, AWS, HBase, Oozie, Kerberos, Ranger

Hadoop Administrator

AstraZeneca,Chennai Jul 2015 to Jan 2018

Job Responsibilities

Handle the installation and configuration of a Hadoop cluster.

Build and maintain scalable data using the Hadoop ecosystem and other open source components like Hive and HBase.

Monitor the data streaming between web sources and HDFS.

Close monitoring and analysis of the Map Reduce job executions on cluster at task level.

Inputs to development regarding the efficient utilization of resources like memory and CPU utilization based on the running statistics of Map and Reduce tasks.

Changes to the configuration properties of the cluster based on volume of the data being processed and performance of the cluster.

Setting up Identity, Authentication and Authorization.

Used DevOps for automation the tasks.

Maintaining Cluster in order to remain healthy and in optimal working condition.

Handle the upgrades and Patch updates.

Set up automated processes to analyze the System and Hadoop log files for predefined errors and send alerts to appropriate groups.

Inputs to development regarding the efficient utilization of resources like memory and CPU utilization.

Based on the running statistics of Map and Reduce tasks.

Balancing HDFS manually to decrease network utilization and increase job performance.

Commission and decommission the Data nodes from cluster in case of problems.

Set up automated processes to archive/clean the unwanted data on the cluster, in particular on Name node and Secondary name node.

Discussions with other technical teams on regular basis regarding upgrades, Process changes, any special processing and feedback.

Environment: Horton works (HDP 2.2), HDFS, Hive, Spark, Sqoop, SQL, Splunk, Linux, Zookeeper, HBase, Oozie.

Tivoli Administrator

HCL-Chennai Apr 2012 to June 2015

Job Responsibilities

Support customer accounts on Backup & Storage technologies.

Planning TSM backups with required retention periods and defining the policy domains and management class accordingly, binding of client’s data to required management class to store the data in predefined storage pools like Disk, sequential storage pools and copy data from primary storage pools to copy storage pool for offsite.

Configuring TSM operations like expiration, migration, reclamation, collocation and media management.

Define and configure the client and administrative schedules, checking the status.

Checking error reports on the servers and health check of the servers, if any recovery LOG or Database or Storage pool related issues, troubleshooting based on the criticality.

Ensure that all Backup server, tape library hardware and software are maintained to current levels, including system firmware code and that all critical hardware and corresponding appropriate software is placed on service/maintenance contracts.

Documentation of infrastructure, software, systems configuration, process and policies.

Detect/diagnose and resolve hardware issues (server, tape library, etc) and interface with vendor, manufacturer for H/W and S/W as necessary.

Creating daily / Monthly reports on backup status for all customer accounts,

Working on EMC Avamar like installation, backup, restore and configuring the policy and schedules.

Working on Data Domain 990, creating and managing NFS and CIFS shares for backup and troubleshooting the issues.

Working on DD OS code upgrade as per the suggestion from Vendor.

Basic knowledge on EMC SAN (Celera) like creating New File systems and exporting NFS and CIFS shares and providing required access.

Basic knowledge on Symantec Net Backup appliance like configuring backup, creating new policies and working on master and media server issues.

Monitoring the Hadoop Clusters.

Worked on Hadoop Basic Failures.

Supported Architecture team in building of Hadoop Clusters.

Worked on BMC Remedy for Hadoop Failures.

MANAGING SERVER:

Checking administrative schedules like DB BACKUP, BACKUP STG.

Configuring SERVER process like Expiration, Storage Pool Migration, Collocation and Reclamation of TAPE Storage Pool Volumes.

Configuring TSM Server scripts to make the administration easy.

MANAGING LIBRARY:

Configuring library and library paths, drives and drive paths, device class.

Audit Library if any mismatch in LIBRARY inventory and TSM inventory.

Maintaining minimum number of scratch volumes.

Tivoli Administrator

IBM India Private Limited - Hyderabad June 2011 to Mar 2012