Resume

Sign in

Sr.Hadoop Admin

Location:
Chicago, Illinois, United States
Posted:
October 27, 2016

Contact this candidate

Resume:

RAJENDRA REDDY

Mail id : acw9pu@r.postjobfree.com

Contact Details: 860-***-****

Professional Summary

Around 7+ years of Information Technology experience. Extensive experience in design Experience in Hadoop administration activities such as installation and configuration of clusters using Apache, Cloudera, Hortonworks, AWS, ECS and ISILON Able to understand business and technical requirements quickly; Excellent communications skills and work ethics; Able to work independently; Experience working with clients of all sizes.

Over 3 years of experience in Hadoop Administration.

Hands on experience in installing, configuring, and using Hadoop ecosystem components like Hadoop HDFS, Yarn, MapReduce, HBase, Oozie, Hive, Sqoop, Pig, Flume, SmartSense, Storm, Kafka, Ranger, Falcon and Knox.

Experience in deploying Hadoop cluster on Public and Private Cloud Environment like Cloudera, Hortonworks, Amazon AWS, RackSpace,ECS & ISILON.

Setting up automated 24x7 monitoring and escalation infrastructure for Hadoop cluster using Nagios and Ganglia.

Experience in managing and reviewing Hadoop log files.

Experience in setting up the High-Availability Hadoop Clusters.

Ability to prepare documents including Technical Design, Testing strategy, and supporting documents.

Hands on experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4), Yarn distributions(CDH 5.X).

Hands on experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Hortonworks (HDP 2.2, HDP2.3).

Hadoop Cluster capacity planning, performance tuning, cluster Monitoring, Troubleshooting.

Good experience on Design, configure and manage the backup and disaster recovery for Hadoop data.

Hands on experience in analyzing Log files for Hadoop and eco system services and finding root cause.

Experience in understanding the security requirements for Hadoop and integrating with Kerberos authentication infrastructure- KDC server setup, creating realm /domain, managing.

Experience on Commissioning, Decommissioning, Balancing, and Managing Nodes and tuning server for optimal performance of the cluster.

As a admin involved in Cluster maintenance, trouble shooting, Monitoring and followed proper backup& Recovery strategies.

Experience in HDFS data storage and support for running map-reduce jobs.

Installing and configuring hadoop eco system like sqoop, pig, hive.

Knowledge on Hbase and zookeeper.

Experience in importing and exporting the data using Sqoop from HDFS to Relational Database systems/mainframe and vice-versa.

Hands on experience on Nagios and Ganglia tool.

Scheduling all hadoop/hive/sqoop/Hbase jobs using Oozie.

Rack aware configuration for quick availability and processing of data.

Handsome experience in Linux admin activities

Good understanding in Deployment of Hadoop Clusters Using Automated Puppet scripts

Experience in hardware recommendations, performance tuning and benchmarking

Experience in IP Management (IP Addressing, Sub-netting, Ethernet Bonding, Static IP)

Flexible with Unix/Linux and Windows Environments working with Operating Systems like Centos 5/6, Ubuntu 10/11 and Sun.

Experience in Linux Storage Management. Configuring RAID Levels, Logical Volumes.

Technical Skills:

Hadoop Framework : HDFS, Map Reduce, Pig, Hive, Hbase, sqoop, zookeeper, Ranger, Storm, Kafka, Oozie, flume, Hue, Knox

Databases : Oracle 9i/10g, DB2, SQL Server, MYSQL

Cloud : AWS, ECS and ISILON

Operating Systems : Linux RHEL/Ubuntu/CentOS, Windows (XP/7/8)

Scripting Languages : Shell scripting

Network Security : Kerberos

Monitoring Tools : Cloudera Manager, Ambari, Nagios, Ganglia

Education and Certifications:

Bachelor of Engineering in Electronics and Communication Engineering.

Cloudera Certified Hadoop Administrator

Professional Experience

Discover Financial Services, Chicago, IL Sep 2015 – Present

Role: Sr Hadoop Administrator

Responsibilities:

Manage several Hadoop clusters in production,development, Disaster Recovery environments.

Responsible for building a cluster on HDP 2.2 and HDP 2.3

Work with engineering software developers to investigate problems and make changes to the Hadoop environment and associated applications.

Expertise in recommending hardware configuration for Hadoop cluster

Installing, Upgrading and Managing Hadoop Cluster on Hortonworks

Trouble shooting many cloud related issues such as Data Node down, Network failure and data block missing.

Major Upgrade from HDP 2.2 to HDP 2.3.

Managing and reviewing Hadoop and HBase log files

Proven results-oriented person with a focus on delivery

Built and configured log data loading into HDFS using Flume.

Performed Importing and exporting data into HDFS and Hive using Sqoop.

Managed cluster coordination services through Zoo Keeper.

Provisioning, installing, configuring, monitoring, and maintaining HDFS, Yarn, HBase, Flume, Sqoop, Oozie, Pig, Hive,Ranger,Falcon,Smartsense,Storm,Kafka.

Recovering from node failures and troubleshooting common Hadoop cluster issues.

Scripting Hadoop package installation and configuration to support fully-automated deployments.

Supporting Hadoop developers and assisting in optimization of map reduce jobs, Pig Latin scripts, Hive Scripts, and HBase ingest required.

Implemented Kerberos for authenticating all the services in Hadoop Cluster.

System/cluster configuration and health check-up.

Continuous monitoring and managing the Hadoop cluster through ambari.

Created user accounts and given users the access to the Hadoop cluster.

Resolving tickets submitted by users, troubleshoot the error documenting, resolving the errors.

Performed HDFS cluster support and maintenance tasks like Adding and Removing Nodes without any effect to running jobs and data.

Environment: Hadoop HDFS, Map Reduce, Hive 10.0, Pig, Puppet, Zookeeper, HBase, Flume, Ganglia, Sqoop, Linux, CentOS, Ambari

Verizon, Syracuse, NY Jul 2014 – Aug 2015

Role: Hadoop Administrator

Responsibilities:

Responsible for implementation and ongoing administration of Hadoop infrastructure.

Aligning with the systems engineering team to propose and deploy new hardware and software environments required for Hadoop and to expand existing environments.

Working with data delivery teams to setup new Hadoop users. This job includes setting

up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users.

Cluster maintenance as well as creation and removal of nodes using Cloudera Manager

Enterprise

Performance tuning of Hadoop clusters and Hadoop MapReduce routines.

Screen Hadoop cluster job performances and capacity planning.

Monitor Hadoop cluster connectivity and security

Configuring of Hive, PIG, Impala, Sqoop, Flume and Oozie in CDH 5

Manage and review Hadoop log files.

File system management and monitoring.

Major Upgrade from cdh 4 to chd 5.2

HDFS support and maintenance.

Collaborating with application teams to install operating system and Hadoop updates,

patches, version upgrades when required.

Scheduling and Managing Oozie Jobs to automate sequence of rotational activity.

Deciding on security and access control model for cluster and data protection.

Testing plan of production cluster before and after hadoop installation for HA's and performance.

Planning on requirements for migrating users to production beforehand to avoid last minute access issues.

Planning on data topology, rack topology and resources availability for users to share as required.

Planning and implementation of data migration from existing staging to production cluster.

Installed and configured Hadoop ecosystem components like MapReduce, Hive, Pig, Sqoop, HBase, ZooKeeper, fuse and Oozie.

Supported MapReduce Programs and distributed applications running on the Hadoop cluster.

Prepared multi-cluster test harness to exercise the system for performance, failover and upgrades.

Ensured data integrity using 'fsck' and another Hadoop system admin tools for block corruption.

Performed a POC on cluster back using distcp, Cloudera manager BDR and parallel ingestion.

Implemented commissioning and decommissioning of data nodes, killing the unresponsive task tracker and dealing with blacklisted task trackers.

Performed various configurations which include networking and IPTables, resolving hostnames, SSH key less login.

Implemented Kerberos Security Authentication protocol for existing cluster.

Configured Ganglia which include installing gmond and gmetad daemons which collects all the metrics running on the distributed cluster and presents them in real-time dynamic web pages which would further help in debugging and maintenance.

Continuous monitoring and managing the Hadoop cluster through Ganglia and Nagios.

Dumped the data from HDFS to MYSQL database and vice-versa using SQOOP.

Environment: Hadoop Hdfs, Mapreduce, Yarn, Hive, Pig, Flume, Oozie, Sqoop, Cloudera Manager.

Syntel Inc, Phoenix, AZ Apr 2012 – Jun 2014

Role: Hadoop Administrator

Description: Syntel, a leading IT Outsourcing, IT Consulting firm offering managed IT services, Enterprise Business Solutions and Collaborative client partnerships through its flexible Global Delivery model. Syntel has never been the biggest. Instead, Syntel focus on select industries and service lines, enabling us to deliver the best possible solutions to meet our clients' needs.

Responsibilities:

Solid Understanding of Hadoop HDFS, Map-Reduce and other Eco-System Projects.

Installation and Configuration of Hadoop Cluster

Working with Cloudera Support Team to Fine tune Cluster

Experienced in managing and reviewing Hadoop log files.

Created user accounts and given users the access to the Hadoop cluster.

Worked with application teams to install operating system and Hadoop updates, patches, version upgrades as required.

Experienced in managing and reviewing Hadoop log files

Advised file system team on optimizing IO for Hadoop / analytics workloads.

Importing the data from the MySql and Oracle into the HDFS using Sqoop.

Importing the unstructured data into the HDFS using Flume.

Written Map Reduce java programs to analyze the log data for large-scale data sets.

Experienced in running Hadoop streaming jobs to process terabytes of XML format data.

Load and transform large sets of structured, semi structured and unstructured data.

Responsible to manage data coming from different sources.

Supported Map Reduce Programs those are running on the cluster.

Assisted with data capacity planning and node forecasting.

Upgraded the Hadoop cluster from cdh3 to cdh4.

Jobs management using Fair scheduler.

Cluster coordination services through Zoo Keeper.

Involved in loading data from UNIX file system to HDFS.

Managing Disk File Systems, Server Performance, Users Creation and Granting file access Permissions and RAID configurations.

Automate administration tasks through the use of scripting and Job Scheduling using CRON.

Manage the day to day operations of the cluster for backup and support.

Creating and managing Logical volumes. Using Java JDBC to load data into MySQL.

Environment: Hadoop Hdfs, MapReduce, Hive, Pig, Flume, Oozie, Sqoop, Cloudera Manager.

Syntel Inc, India July 2010 – Mar 2012

Role: Linux/MySQL Administrator

Description: Syntel, a leading IT Outsourcing, IT Consulting firm offering managed IT services, Enterprise Business Solutions and Collaborative client partnerships through its flexible Global Delivery model. Syntel has never been the biggest. Instead, Syntel focus on select industries and service lines, enabling us to deliver the best possible solutions to meet our clients' needs.

Responsibilities: Linux Administration

Environment Set-up:

Installation and configuration of Linux for new build environment.

Created Virtual server on Citrix Xen Server based host and installed operating system on Guest Servers.

Installed Pre-Execution environment boot and Kick start method on multiple servers, remote installation of Linux using PXE boot.

Installation and Set-up:

Software installation, disk partitioning, file system creation, user id creation and configuration of Linux.

Configuring and Managing Yum Repository

System Management:

Disk space Management, Disk quota management, Maintenances of password & shadow file, NIS master and client configuration, NFS file system configuration.

Working on Logical Volume Manager (LVM).

Installed and Configured 5 node Hadoop cluster

LVM configuration and increasing the size of Logical Volume and updating the file system

Security (Internal/External) :

Restricted file and directory access permissions securely by set UID & GID. Set as per

project requirements and data security.

IP tables configuration and Maintenance.

Performed various configurations which include networking and IP Tables, resolving hostnames, SSH key less login.

User & Group Management:

User Creation, Modification & Deletion as per requirements.

Group id creation, deletion, and addition of a group for a particular user.

Data Integrity and Contingency:

Utilization of dump and restore for file system backup and restoration.

Log management using cron jobs

Automate administration tasks through use of scripting and Job Scheduling using CRON.

Performance tuning of MySQL engines like MYIASM and Innodb.

My sql application using MySQL workbench, Toad for MySQL, MySQL Administrator.

Setting up MySQL Cluster on 2 node servers.

Performance tuning for high transaction and volumes data in mission critical environment.

Environment: MySQL, PHP, Shell Script, Apache, Linux.

Secugen Inc. India May 2009 – Jun 2010

Role: Jr. MySQL Administrator

Description: Secugen Corporation is the world's leading provider of advanced, optical fingerprint recognition technology, products, development tools and platforms. Secugen has been serving the global biometrics industry since 1998 and strives to provide its customers with the highest quality products and service through continuous research & development and dedicated technical support.

Responsibilities:

Experience in MYSQL installation, configuration and migration

Performed installation and testing of database versions and software products in coordination with systems support teams.

Supported SQL queries and databases by updating and tuning in proper manner

Experience in Installation, configuration and maintenance of MSSQL Cluster.

Resolved database related problems by troubleshooting and evaluation of database events.

Define requirements for physical database consolidation, backup and recovery and application database disaster recovery.

Set up and managed MySQL Master to Master replication, Master to Slave replication and MySQL Sharding.

Suggested improvement processes for increasing efficiency of applications and current processes

Maintained system documentation relating to application architecture, assembling and technical platforms.

Provided technical assistance on installation and maintenance of database software components and related upgrades.

Coordinated with external vendors for tracking of routine incidents and operational problems.

Synchronization of replicated database.

Benchmarking of database and servers.

Implemented MSSQL codes including procedures, functions, and triggers.

Collaborated with system teams and development teams for codes release including MySQL database changes and patches.

On-call support to users.

ENVIRONMENT: MySQL, PHP, Shell Script, Apache, Linux.



Contact this candidate