*

Dilip Kumar Pandeti

Senior Hadoop Administrator

Contact: +91-903*******, LinkedIn: https://www.linkedin.com/in/varmadilip Summary of Qualification

• Good knowledge and experience on Linux, AWS and Bigdata Technologies

• Extensive experience on Upgrading Ambari and HDP versions

• Extensive experience on managing large clusters that are having around 600 Nodes

• Extensive experience on performing administration, configuration management, monitoring, debugging in Hadoop Clusters

• Expertise in Ansible and Chef configuration management tool

• Expertise in adding Hadoop components using Ambari

• Daily ticket analysis of open and critical operations issues

• Hands on experience in resolving complex technical issues like recovery of nodes, Maintenance of Hadoop configuration files.

• Cluster maintenance as well as creation and removal of nodes.

• Good knowledge in AWS services such as EC2, Simple Storage Service(S3)

• Having good experience on Nagios,Grafana,Prometheus Monitoring Tools

• Having good experience on Writing the Bash Scripts and JQ

• Excellent Interpersonal Skills, Communication skills, documentation skills, problem solving ability, Quality conscious and multi-tasked environment

• Participating in on-calls with on-site team and with the project managers to discuss the ongoing Hadoop activities

• Setting up password less authentication on Linux servers

• Expertise in Ranger to apply security policies for Hadoop services

• Good knowledge on Jenkins

• Communication with Database,application,Network, Infrastructure teams for Hadoop

• Support to users to meet their SLA’s

Skills

Programming Language : Shell Script, Json query language Databases : MySQL

Configuration Management : Chef,Ansible

Frameworks : Hortonworks, Cloudera, Amazon Web Services (EC2, S3, IAM) Monitoring Tools : Grafana,Prometheus,Nagios,Ambari Defect Tracking Tools : Apple Centralstation, NmSys, Jira & PagerDuty 2

OS Environment : Linux (Ubuntu, Centos, Redhat, Amazon linux, Oracle Linux) & Windows

Version Control : Github(Bitbucket)

Certifications :

RedHat

• RHCSA (Redhat Certified System Administrator) (EXAM CODE : EX200)

• RHCE (Redhat Certified Engineer) (EXAM CODE : EX300) Certification number : 120-203-096

Verification url : https://www.redhat.com/rhtapps/verify/?certId=120-203-096 Hadoop

• Cloudera Certified Hadoop Administrator (CDH5) (EXAM CODE : CCA-500) Certification number : 100-012-295

Verification url : http://certification.cloudera.com/verify Experience Details

Rakuten

Period : Sep 2019 to Till Date

Role : SENIOR SYSTEMS ENGINEER II (Hadoop administrator - Hortonworks) Description:

Rakuten is a Japanese electronic commerce and online retailing company based in Tokyo. It’s one of the largest e-commerce sites in Japan, It has more than 70 Services operating through Internet Services, FinTech and Mobile. Most of these services Data in Hadoop to process and analyse it for business requirements.

My day in a life as Hadoop Administrator:

• Onboard users, create keytabs and ranger policies, Adding users to groups,Delete Inactive users

• Construct Kerberos node (KDC & Kadmin),Migrate Kadmin server

• Create VIP by LBaaS (Load balancer as a service)

• Create Grafana dashboard for a new cluster, Set alarm using Grafana

• Setting up kerberos cross realm configuration

• Create and adjust queues, add users to Queue ACL

• Commission, Decommission nodes

• Handle disk issues on Hadoop slave nodes and replacing failed disks

• Handle Hardware and memory related issues on Hadoop nodes

• Adding new ssd/hdd to Hadoop slave nodes

3

• Restart Nodemanager service due to GC issue

• Running HDFS balancer

• Creating HDFS snapshots to users

• Cleaning HDFS corrupt files

• Create TDE (transparent data encryption) zone for user

• Creating HDP2 cluster with Chef and HDP3 cluster with Ambari

• Setting up Xinclude servers for client configuration

• Handle LevelDB issues for Timeline service

• Recover unhealthy journal node

• Fix device name change issues for AMD CPU Dell Nodes

• Setup Local repository for HDP3

• Setup Ambari server with SSL

• Automatic sending notification about HDFS small size files to avoid Namenode issue

• Proactive monitoring of Hadoop Clusters

• Providing quick support to User inquiries

Value adds:

• Created bash script to notify Datanode DFS usage > 90% to avoid slowness issue

• Created bash script to identify and restart Nodemanagers that are not performing GC as expected to avoid OOM issue

• Found a way to add Datanode service without impacting HDP3 Ambari cluster

• Configured Hive LLAP with SSD cache to improve Query performance

• Configured Namenode QOS on Prod clusters

• Prepared RestAPI for SSL enabled Ambari

• Created packet loss graph using Prometheus and Grafana to detect network issue of slave nodes to avoid slowness issue

• Providing training to users about Rakuten Hadoop

• Fixed Cloudera subscription issue to use Local repositories for PROD HDP3 clusters Customer Satisfaction:

• The outcome of the above mentioned value Adds made the customers happy because of the following reasons.

• Datanode DFS usage script avoid slowness outage

• Automation of Nodemanager restart saved lot of manual time(2Hours per week) and avoid Nodemanager going down due to OOM

• Datanode commission at HDP3 avoided restarting of services

• Hive LLAP with SSD cache improved query performance

• Packet loss graph is useful to monitor network issue Wipro Limited

Customer : Apple (www.apple.com)

4

Period : Apr 2018 to Sep 2019

Role : Hadoop Administrator(Hortonworks)

Description:

Apple has lot of own products like iTunes, iPhones, iCloud, iCloud Mail, iwork, News, Maps, FaceTime, GameCenter, Central Station, iPad and iMessage etc. Apple is using BigData for storing and analyzing the data to produce accurate results and providing support to the customers. Apple has physical and cloud servers to store the data. We are giving infrastructure support for Apple devices/applications. Particularly we are managing the data related to one of the Apple application called iTunes. As part of the data is stored in HDFS, so we are providing end to end support to make the application more reliable. My day in a life as Hadoop Administrator:

• Setting up HDP Cluster based on the client requirement

• Oracle Enterprise Linux patching on Hadoop Clusters

• Creating Change Requests, Incidents, Tasks on centralstation for Hadoop related issues

• Troubleshooting yarn,hdfs, hbase, sqoop, oozie hardware related isslues like mount point issues, host unreachable

• Creating Named users and EDW users using ansible yml files

• Commissioning and De-commissioning Hadoop Datanodes

• Writing ansible playbooks, ansible ad-hoc commands and running them using Ansible Tower

• Kerberos principles and keytab creation for application users

• Creating name space quota and disk space quota for hadoop users

• Adding yarn queues on hadoop cluster

• Actively participating on weekly meeting calls and working on on-calls Value Adds:

• Wrote a bash script and put it in cronjob to find out the hadoop long running jobs which are running more 6 Hours and kill them

• wrote script to send mail notification when each datanode dfs utilization reaches to > 70%

• Wrote a bash script and put it in cronjob to find out hadoop log files which are not accessed more than 90 days and to delete them and to send notification as well. Customer Satisfaction:

• The outcome of the above mentioned value Adds made the customers happy because of the following reasons.

• Scripts almost minimized the manual work from 2 or 3 hours to < 10 minutes.

• Scripts can be reusable and can do the modifications whenever required. Valuelabs Solutions

5

Customer : Benefitfocus (www.benefitfocus.com)

Period : Nov 2016 – Jan 2018

Role : Hadoop Administrator(Hortonworks)

Description:

Benefitfocus is the largest healthcare benefits software provider in the U.S. Benefitfocus knows healthcare. It's technology solutions reduce costs, provide market advantage, manage risk and improve consumer satisfaction. Benefitfocus products and services give a competitive edge to carriers, employers and brokers as they help consumers manage complex decisions in an ever- changing healthcare benefits environment.

My day in a life as Hadoop Administrator:

• Adding a new node to an existing cluster and Decommissioning a node using Ambari

• Upgrading HDP and Ambari to new versions

• Adding Hadoop services to a cluster using Ambari

• Changing configuration of services using Ambari

• Creating a home directory for user and configure permissions

• Troubleshooting Hadoop related issues using application log files

• Configuring and managing alerts

• Troubleshooting oozie failed jobs

• Copying data between two clusters using distcp

• Install and configure Hadoop services like Storm, Scala

• Writing Ansible playbooks to automate the tasks

• Providing ACLs to the Hadoop directories

• Migrating Hadoop production mysql server to new mysql server

• Downgrading mysql jar files to specific version

• Writing Firewall rules (IPTABLLES) in linux server

• OpenLDAP Installation and configuration

• Adding OpenLDAP users

• Performance tuning of Hadoop Clusters

• Providing Security policies for HDFS, HIVE, HBASE by using Ranger Value Adds:

• Wrote bash scripts to automate MySQL Databases backups and set it in the crontab to run in the specific time

• Wrote bash script to remove the old hadoop log files.

• Wrote Ansible scripts for below operations to save time & eliminate manual work

• To copy repository files to new datanodes during new datanodes installation

• To create Hadoop Directories in new datanodes

• To copy customized jar files to specific directories in new datanodes

• To create softlinks

• To change the ownership and permissions for Hadoop directories Customer Satisfaction:

6

• The outcome of the above mentioned value Adds made the customers happy because of the following reasons.

• Scripts almost minimized the manual work from 2 or 3 hours to < 10 minutes.

• Scripts can be reusable and can do the modifications whenever required. Conferra software solutions

Customer : Share This, Inc (www.sharethis.com)

Period : Sep 2014 – Oct 2016

Role : Hadoop Administrator(Cloudera)

Description:

ShareThis, Inc. provides a social sharing platform and solutions for advertisers/publishers. The project deals with administration, maintenance and support of ShareThis database and data warehousing systems.

My day in a life as Hadoop Administrator:

• Administration & Monitoring Hadoop Clusters on EC2

• Replacing Hadoop nodes and recover the Hadoop Daemons.

• Balancing the data on clusters.

• Adding New user on Hadoop

• Having good knowledge on Kerberos security

• Installation and administration of Cloudera Hadoop distribution and its echo systems.

• Adding & Monitoring Services in Nagios.

• AMI Generation and Migrate the mobile website from classic to VPC.

• Production support on AWS Cloud.

• Maintain Github repos for Configuration Management.

• Fixing on going incidents.

• Automation of s3 files backup and remove.

• Taking AWS security groups backup upload to s3 and its automation.

• Chef administration, launching services using chef recipes.

• Transfer data to and from cluster, using Sqoop and various storage media such as MySQL tables

• Providing VPN user accounts and managing their accounts.

• Adding nagios plugins.

• Part of 24 * 7 on call rotations.

Value Adds:

• Automated bash scripts to delete unwanted AWS-S3 Buckets.

• Wrote the bash script to filter the specific properties of AWS-EC2 instances like Security group, Elastic IP, Public DNS, Instance Type, Region Name Etc. Customer Satisfaction:

7

• The outcome of the above mentioned value Adds made the customers happy because of the following reasons.

• Scripts almost minimized the manual work from 1 hour to < 10 minutes.

• Scripts can be reusable and can do the modifications whenever required. Customer : Techmileage, Inc (www.techmileage.com)

Period : Mar 2014 – Aug 2014

Role : Hadoop Administrator

Description:

Digital Caddies Players Network is a unique GPS mobile golf information system that thousands of players rely on for course information during play. Using high quality tablets that support video streaming through high-speed 3G wireless networks, players access details on each hole, track their score, and enjoy many additional benefits. It’s like having a digital concierge right in the cart with them throughout the round, so they use the equipment throughout every round of golf. Each tablet is securely mounted in the cart, where both occupants can clearly see the entire screen.

My day in a life as Hadoop Administrator:

• MYSQL basic administration (Backups, User Privileges etc.)

• Nagios & Pager duty setup.

• Administered Linux-based servers, monitoring performance and ensure system integrity

• Design, develop, and manage data on Hadoop cluster

• End-to-end performance tuning of Hadoop clusters

• Bigdata analysis, BI reports generation using DBVisualizer

• Linux user management

• Wrote bash scripts to setup the single node apache hadoop cluster

Contact this candidate