Sr. Hadoop Administrator Cloudera Operations

Kalyan Yalagandula

https://www.linkedin.com/in/kalyan-a-768b46116/

Sr. Hadoop Administrator – Cloudera Operations and Support

601-***-**** ️ ******.******@*****.***

Professional Summary:

Experienced IT professional with 14 years of expertise in designing, implementing, and managing enterprise-grade technology systems. Specialized in Hadoop administration, Linux administration, and Cloudera Data Platform (CDP) environments across on-prem and cloud infrastructures.

10 years of experience in Hadoop Administration and Big Data technologies with cloud platforms (Azure with Insight, Google Cloud, and AWS), and 4 years of experience in Linux Administration.

Experience with the complete Software Development Lifecycle (SDLC) including design, development, testing, and implementation of moderately to highly complex systems.

Hands-on experience in installation, configuration, support, and management of Hadoop Clusters using Hortonworks, Cloudera, and Apache Hadoop 2.6.

Proficient in Hadoop cluster capacity planning, performance tuning, cluster monitoring, and troubleshooting.

Skilled in designing Big Data solutions for traditional enterprise businesses.

Excellent command over creating backup and recovery procedures and implementing disaster recovery strategies for both offline and online backups.

Involved in benchmarking Hadoop/HBase cluster file systems for various batch jobs and workloads.

Prepared Hadoop clusters for development teams working on POCs (Proof of Concepts).

Experience in minor and major upgrades of Hadoop and its ecosystem components.

Proficient in monitoring and troubleshooting Linux issues related to memory, CPU, OS, storage, and networking.

Hands-on experience analyzing log files for Hadoop and ecosystem services to identify root causes.

Expertise in commissioning, decommissioning, balancing, and managing nodes, as well as tuning servers for optimal cluster performance.

Involved in cluster maintenance, troubleshooting, monitoring, and implementing proper backup and recovery strategies.

Good experience in setting up Linux environments: configuring passwordless SSH, creating file systems, disabling firewalls, setting swappiness, configuring SELinux, and installing Java.

Skilled in planning, installing, and configuring Hadoop clusters using Cloudera and Hortonworks distributions.

Experience installing and configuring Hadoop ecosystem tools like Pig and Hive.

Hands-on experience in installing, configuring, and managing Hue and Apache Atlas (Catalog).

Experience in importing/exporting data using Sqoop from HDFS to relational databases/mainframes and vice versa in enterprise data lake environments.

Experience in importing/exporting logs using Flume.

Skilled in optimizing performance of HBase, Hive, and Pig jobs.

Hands-on experience with Zookeeper and ZKFC for managing and configuring NameNode failover scenarios.

Strong experience in Linux administration activities on RHEL and CentOS.

Experience in deploying Hadoop 2.0 (YARN) architecture.

Familiar with writing Oozie workflows and job controllers for job automation.

Educational Details:

Kakatiya University: Bachelor’s in computer science (India).

International Technological University: MS in software engineering (USA).

Technical Skills:

Category

Tools / Technologies

Hadoop Ecosystem Tools

MapReduce, HDFS, Pig, Hive, HBase, Sqoop, ZooKeeper, Oozie, Hue, Storm, Kafka, Spark, Flume

Programming Languages

Java, Core Java, HTML, C, C++, PL/SQL

Operating Systems

Linux (RHEL, Ubuntu), OpenSolaris, AIX

Scripting Languages

Shell Scripting, HTML Scripting

Cloud Technologies

AWS, Google Cloud Platform (GCP), Microsoft Azure

Cluster Management Tools

HDP Ambari, Cloudera Manager, Hue

Professional Experience:

Client: State Farm, Tempe, AZ Contract April 2020 to till date

Title: Sr. Hadoop Administrator / Support Engineer

Responsibilities:

•Currently working as admin on Cloudera (CDP 7.3.1) distribution for 12 clusters ranging from POC to PROD.

•Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning Data nodes, Troubleshooting, Managing and reviewing data backups, and managing & and reviewing log files.

•Day-to-day responsibilities include solving developer issues, and deployments moving code from one environment to another environment, providing access to new users, and providing instant solutions to reduce the impact of documenting the same and preventing future issues.

•Adding/installation of new components and removal of them through Cloudera Manager.

•Performance tuning on Hue and Hive, impala, and Spark jobs

•Monitored workload, job performance, and capacity planning using Cloudera Manager.

•Interacting with Cloudera support logging the issues in the Cloudera portal and fixing them as per the recommendations.

•Cluster provisioning depends upon the user request and adding gateway nods to the cluster (CRMT tool)

•Retrieved data from HDFS into relational databases with Sqoop. Parsed cleansed and mined useful and meaningful data in HDFS using Map-Reduce for further analysis.

•Installed and configured Apache Hadoop cluster and Flink service on POC clusters

•Implemented custom interceptors for flume to filter data and defined channel selectors to multiplex the data into different sinks.

•Partitioned and queried the data in Hive for further analysis by the BI team.

•Involved in extracting the data from various sources into Hadoop HDFS for processing.

•Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database, and Sqoop.

•Experienced in using Kafka as a distributed publisher-subscriber messaging system.

•Creating and truncating HBase tables in hue and taking backup of submitter ID(s).

•Configuring and managing permissions for the users in hue.

•Responsible for building scalable distributed data solutions using Hadoop.

•Commissioned and Decommissioned nodes on CDH5 Hadoop cluster on red hat LINUX.

•Involved in loading data from the LINUX file system to HDFS.

•Implemented test scripts to support test-driven development and continuous integration.

•Worked on tuning the performance of Pig queries.

•Experience in configuring the Storm in loading the data from MYSQL to HBASE using JMS.

•Responsible for managing data coming from different sources.

•Involved in loading data from UNIX file system to HDFS.

•Experience in managing and reviewing Hadoop log files.

•Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

•Installed Oozie workflow engine to run multiple Hive and Pig jobs.

•Analyzed large amounts of data sets to determine the optimal way to aggregate and report on them.

•Supported in setting up the QA environment and updating configurations for implementing scripts with Pig and Sqoop.

•Ensure high availability and fault tolerance in cloud infrastructure.

•Provide me a Splunk configuration that collects all Hadoop logs and integrates them into a Splunk dashboard

•Optimize resource utilization and cost-effectiveness through auto-scaling and right-sizing strategies.

•Deploy and manage containerized applications using ECS and Lambda for serverless computing.

Environment: HDFS, Map Reduce, Hive, Hue, Pig, Flume, Oozie, Sqoop, Cdh5, Apache Hadoop Spark, Solr, Storm, Cloudera Manager, Red hat, informatica MySQL, and Oracle.

Client: Walmart Labs INC, AR Contract Jan 2019 to Mar 2020

Title: Hadoop admin & support engineer

Responsibilities:

Experienced as a Hadoop admin & and support engineer in Horton Works (HDP 2.6.3) distribution clusters.

Monitor Hadoop cluster connectivity and security.

Manage and review Hadoop log files, File system management and monitoring

HDFS support and maintenance.

Hadoop migration refers to transferring data, informatica ETL processes, and analytics workloads from legacy systems (like traditional data warehouses) to a Hadoop-based ecosystem.

Experienced in production support which involves solving user incidents varies from P1 to P3 depending upon client service level agreement.

Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning of Data nodes, Troubleshooting, Managing & and reviewing log files.

Working with data delivery teams to set up new Hadoop users. This job includes setting up Linux users, setting up Kerberos principals, and testing HDFS, Hive, Spark and Map Reduce access for the new users.

Involved in analyzing system failures, identifying root causes, and recommending courses of action.

Ensure that the Hadoop cluster is up and running all the time.

Monitoring the cluster connectivity and performance

Injecting Hadoop clusters from the enterprise data from data lakes

Manage and review Hadoop log files.

Optimizing ETL Informatica workflows and mappings to leverage Hadoop's parallel processing capabilities, enhancing performance and scalability.

Backup and recovery tasks

Supporting on Microsoft Azure HDInsight clusters

Deploy new clusters on Azure virtual machines by using cloud break.

Providing the server support and maintenance of Azure clusters.

Installing and upgrading, configuring, and maintaining Hadoop eco components on Azure servers

Resource and security management

•Troubleshooting application errors and ensuring that they do not occur again.

•Cluster maintenance as well as creation and removal of nodes using Hadoop Management Admin tools like Ambari.

•Collaborating with application teams to install the operating system and Hadoop updates, patches, and version upgrades when required.

•Experienced in using Kafka as a distributed publisher-subscriber messaging system.

•Import and export data back-end Tera data to Hadoop and vice versa Hadoop to Tera data by use of Sqoop.

•Install and upgrade Python packages on edge nodes.

•Monitoring the hive jobs and number of connections on the hive met store through Grafana web UI

•Hive performance tuning.

• Experienced in Ranger Centralized security administration to manage all security-related tasks in a central UI or using REST APIs.

•Enhanced support for different authorization methods – By use of Ranger Role-based access control.

•Centralize ranger auditing of user access and administrative actions (security-related) within all the components of Hadoop.

•Cloud Solution Design & Implementation: Design and implement scalable, reliable, and secure cloud solutions leveraging GCP services such as Dataproc, Compute Engine, Kubernetes Engine, Cloud Storage, and BigQuery.

•Hadoop/Dataproc Administration: Set up, configure, and manage Hadoop/Dataproc clusters, including scaling nodes, performance tuning, and ensuring high availability.

•Infrastructure Automation: Utilize Terraform and Google Cloud Deployment Manager to automate provisioning, deployment, and lifecycle management of GCP resources.

•CI/CD & Automation Pipelines: Develop and maintain automation pipelines for application deployment, infrastructure provisioning, and environment consistency.

•Monitoring & Performance Optimization: Use Stackdriver (Cloud Logging/Monitoring) to track infrastructure health, troubleshoot issues, and optimize system performance.

•Incident Management & Troubleshooting: Diagnose and resolve cloud infrastructure, networking, and application-related issues in a timely manner.

•Scripting & Automation: Automate repetitive administrative tasks using scripting languages such as Python, Bash, or PowerShell.

•System & Security Administration: Perform Linux system administration, enforce networking best practices, and ensure compliance with cloud security principles.

•Collaboration & Governance: Work closely with developers, operations, and security teams to define cloud architecture best practices and maintain compliance with organizational standards.

•Continuous Learning: Stay updated with the latest GCP services, security guidelines, and industry best practices to enhance cloud solutions.

Environment: HDFS, Map reduce, yarn, zookeeper, Hive, Hue, Pig, Oozie, Sqoop, Apache Hadoop Spark,

Client: State Farm, Tempe, AZ Contract Jan 2018 to Dec 2018

Title: Sr. Software Engineer (Hadoop infrastructure analyst)

Responsibilities:

•Experienced as an admin on Cloudera (CDH 5.13.2) distribution for 12 clusters ranging from POC to PROD. And Horton Works (2.5.3)

•Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning Data nodes, Troubleshooting, Managing and reviewing data backups, and managing & and reviewing log files.

•Day-to-day responsibilities include solving developer issues, deployments moving code from one environment to another environment, providing access to new users and providing instant solutions to reduce the impact of documenting the same, and preventing future issues.

•Adding/installation of new components and removal of them through Cloudera Manager.

•Collaborating with application teams to install the operating system and Hadoop updates, patches, and version upgrades.Designed and managed apache airflow DAGs for automated job scheduling and failure recovery.

•Imprt and export the data from informatica SQLto Hadoop By use of sqoop.

•Worked on the journal node sync and Name node high availability and yarn, hue, and oozie, and Hbase and impala high availability.

•Experienced in Ranger and Atlas integration. and provide the atlas and ranger role-based policies.

•Performance tuning on Hue and Hive, impala, and Spark jobs

•Monitored workload, job performance, and capacity planning using Cloudera Manager.

•Interacting with Cloudera support logging the issues in the Cloudera portal and fixing them as per the recommendations.

•Cluster provisioning depends upon the user request and adding gateway nods to the cluster (CRMT tool)

•Retrieved data from HDFS into relational databases with Sqoop. Parsed cleansed and mined useful and meaningful data in HDFS using Map-Reduce for further analysis.

•Implemented custom interceptors for flume to filter data and defined channel selectors to multiplex the data into different sinks.

•Partitioned and queried the data in Hive for further analysis by the BI team.

•Involved in extracting the data from various sources into Hadoop HDFS for processing.

•Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database, and Sqoop.

•Sentry installation and configuration

•Experienced in using Kafka as a distributed publisher-subscriber messaging system.

•Creating and truncating HBase tables in hue and taking backup of submitter ID(s).

•Configuring and managing permissions for the users in hue.

•Responsible for building scalable distributed data solutions using Hadoop.

•Commissioned and Decommissioned nodes on CDH5 Hadoop cluster on Redhat LINUX.

•Involved in loading data from the LINUX file system to HDFS.

•Implemented test scripts to support test-driven development and continuous integration.

•Worked on tuning the performance of Pig queries.

•Worked with application teams to install operating system, Hadoop updates, patches, and version upgrades as required.

•Experience in configuring the Storm in loading the data from MYSQL to HBASE using JMS.

•Responsible for managing data coming from different sources.

•Involved in loading data from UNIX file system to HDFS.

•Experience in managing and reviewing Hadoop log files.

•Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

•Installed Oozie workflow engine to run multiple Hive and Pig jobs.

•Analyzed large amounts of data sets to determine the optimal way to aggregate and report on them.

•Supported in setting up the QA environment and updating configurations for implementing scripts with Pig and Sqoop.

Environment: HDFS, Map Reduce, Hive, Hue, Pig, Flume, Oozie, Sqoop, Cdh5, Apache Hadoop Spark, Solr, Storm, Cloudera Manager, Red hat, MySQL, and Oracle.

Client: All State, IL Contract April 2017 to Dec 2017

Title: Sr. Hadoop Operations & Administrator

Responsibilities:

•Experienced as admin in Horton works (HDP 2.5.1) distribution for 5 clusters ranging from POC to PROD.

•Cluster capacity planning depends upon the data usage.

•Designed and configured the Baston-Edge node configuration.

•Designed and configured HA of Hive & HBase services.

•Fine-grained authorization to do a specific action and/or operation with a Hadoop component/tool and managed through a central administration Ranger tool.

•Experienced in managing and reviewing log files. (Identify the max backup index and max backup size of Log4j properties of all services in Hadoop) and integrated the Hadoop log4j properties of AWS Cloud Watch.

•Experienced on use to dump the data from Hadoop to AWS S3 buckets through the access key and secret key.

Experienced on a setup spark service on AWS EMR cluster on EC2 instances (S3 nodes).

•Changed the zookeeper and journal node edit directories (zookeeper and journal nodes have multiple directories)

•Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning Data nodes, Troubleshooting, Managing and reviewing data backups, and managing & and reviewing log files.

•Day-to-day responsibilities include solving developer issues, deployments moving code from one environment to another environment, providing access to new users providing instant solutions to reduce the impact documenting the same, and preventing future issues.

•Experienced in adding/installing new components and removal of them through Ambari.

•Monitoring systems and services through the Ambari dashboard to make the clusters available for the business.

•Architecture design and implementation of deployment, configuration management, backup, and disaster recovery systems and procedures.

•Hand-on experience on cluster upgradation and patch upgrade without any data loss and with proper backup plans.

•Changing the configurations based on the requirements of the users for the better performance of the jobs.

•Experienced in Ambari-alerts (critical & and warning) configuration for various components and managing the alerts.

•Provided security and authentication with ranger where ranger admin provides administration and user sync adds the new users to the cluster.

•Good troubleshooting skills on Hue, which provides GUI for developers/business users for day-to-day activities.

•Developed Map Reduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.

•Implemented complex Map Reduce programs to perform joins on the Map side using distributed cache.

•Implemented Name Node HA in all environments to provide high availability of clusters.

•Involved in snapshots and mirroring to maintain the backup of cluster data and even remotely.

•Experienced in managing and reviewing log files. (Identify the max backup index and max backup size of Log4j properties of all services in Hadoop)

•Helping the users in production deployments throughout the process.

•Experienced in production support which involves solving user incidents varies from sev1 to sev5.

•Managed and reviewed Log files as a part of administration for troubleshooting purposes. Communicate and escalate issues appropriately.

•As an admin followed standard Backup policies to ensure the high availability of the cluster.

•Documented the systems processes and procedures for future reference.

•Worked with the systems engineering team to plan and deploy new environments and expand existing clusters.

•Monitored multiple cluster environments using AMBRI Alerts, Metrics

Environment: HDFS, Yarn, Map reduce, pig, zookeeper, spark, Kafka, Horton works, Ambari.

Client: Synchrony Financial, IL Contract June 2016 to Mar 2017

Title: Software Engineer (Hadoop).

Responsibilities:

•Experienced as admin in Horton works (HDP 2.2.4.2) distribution for 6 clusters ranging from POC to PROD.

•Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning Data nodes, Troubleshooting, Managing, and reviewing data backups, and managing & and reviewing log files.

•Day-to-day responsibilities include solving developer issues, and deployments moving code from one environment to another environment, providing access to new users, and providing instant solutions to reduce the impact of documenting the same and preventing future issues.

•Experienced in adding/installing new components and removal of them through Ambari.

•Monitoring systems and services through the Ambari dashboard to make the clusters available for the business.

•Architecture design and implementation of deployment, configuration management, backup, and disaster recovery systems and procedures.

•Hand-on experience on cluster upgradation and patch upgrade without any data loss and with proper backup plans.

•Changing the configurations based on the requirements of the users for the better performance of the jobs.

•Experienced in Ambari-alerts configuration for various components and managing the alerts.

•Provided security and authentication with ranger where ranger admin provides administration and user sync adds the new users to the cluster.

• Experience in Standardize authorization Ranger Acees method across all Hadoop components.

•Good troubleshooting skills on Hue, which provides GUI for developers/business users for day-to-day activities.

•Developed Map Reduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.

•Implemented Name Node HA in all environments to provide high availability of clusters.

•Capacity scheduler implementation in all environments to provide resources based on the allocation.

•Create queues and allocate the cluster resources to provide priority for jobs.

•Experienced in Setting up the project and volume setups for the new projects.

•Involved in snapshots and mirroring to maintain the backup of cluster data and even remotely.

•Implementing the SFTP for the projects to transfer data from External servers to servers.

•Experienced in managing and reviewing log files.

•Working experience in maintaining MySQL database creation setting up the users and maintaining the backup of cluster metadata databases with corn jobs.

•Setting up MySQL master and slave replications and helping business applications maintain their data in MySQL Servers.

•Helping the users in production deployments throughout the process.

•Experienced in production support which involves solving user incidents varies from sev1 to sev5.

•Managed and reviewed Log files as a part of administration for troubleshooting purposes. Communicate and escalate issues appropriately.

•As an admin followed standard Backup policies to ensure the high availability of the cluster.

•Involved in Analyzing system failures, identifying root causes, and recommending courses of action.

•Documented the systems processes and procedures for future reference.

•Worked with the systems engineering team to plan and deploy new environments and expand existing clusters.

•Monitored multiple cluster environments using AMBRI Alerts, Metrics, and Nagios.

Environment: Hadoop HDFS, Map Reduce, Hive, Pig, Flume, Oozie, Sqoop, Eclipse, Horton works, and Ambari.

Client: Cigniti Technologies-HYD-IND Contract Sep 2012 to April 2014

Title: Sr. Linux System Engineer

Responsibilities:

•Installation and configuration of Linux for the new build environment.

•Created a Virtual server on Citrix Xen server-based host and installed operating system on Guest Servers.

•Configuring NFS, DNS.

•Updating YUM Repository and Red Hat Package Manager (RPM).

•Created RPM packages using RPMBUILD, verifying the new build packages and distributing the package.

•Configuring distributed file systems administering NFS server and NFS clients and editing auto-mounting mapping as per system/user requirements.

•Installation, configuration, and maintenance of FTP servers, NFS, RPM, and Samba.

•Configured SAMBA to get access to Linux-shared resources from Windows.

•Created volume groups logical volumes and partitions on the Linux servers and mounted file systems on the created partitions.

•Deep understanding of monitoring and troubleshooting mission-critical Linux machines.

•Experience with Linux internals, virtual machines, and open-source tools/platforms.

•Improve system performance by working with the development team to analyze, identify, and resolve issues quickly.

•Ensured data recoverability by implementing system and application-level backups.

•Performed various configurations which included networking and Iptables, resolving hostnames, and SSH keyless login.

•Managing Disk File Systems, Server Performance, user creation, and Granting file access Permissions and RAID configurations.

•Support pre-production and production support teams in the analysis of critical services and assist with maintenance operations.

•Automate administration tasks using scripting and Job Scheduling using CRON.

Environment: Linux, Citrix Xen Server 5.0, veritas volume manager, and net backup.

Client: People Link, HYD-IND Contract Jan 2010 to Aug 2012

Title: Linux Administrator

Responsibilities:

•Patched RHEL5 and Solaris 8, 9, and 10 servers for EMC Power Path Upgrade for VMAX migration.

•Configured LVM (Logical Volume Manager) to manage volume group, logical and physical partitions, and importing new physical volumes.

•Maintained and monitored all servers' operating system and application patch level, disk space and memory usage, user activities daily, administration on Sun Solaris and RHEL systems, and management archiving.

•Installed, configured, troubleshooted, and maintained Linux Servers and Apache Web servers, configured and maintenance of security and scheduling backups, and submitted various types of corn jobs.

•Installations of HP Open view, a monitoring tool, in more than 300 servers and worked with monitoring tools such as Nagios and HP Open view.

•Creation of VMs, cloning, and migrations of the VMs on VMware vSphere 4.0/4.1

•Setup and configured Apache to integrate with IBM WebSphere in load balancing environment.

•RHEL 4.1, Red Hat Linux 8/9, IBM x series and HP ProLiant, Windows.

•Installing and upgrading OE & Red Hat Linux and Solaris 8/9/10 x86 & SPARC on Servers like HP DL 380G3, 4, and 5 & and Dell Power Edge servers.

Environment: Linux 5/4, Sun Solaris 10/9/, Oracle 10g, SUN Servers, SUN Fires, Linux, HP open view service desk (OVSD), Kick start, Jump Start, Fujitsu Prime power servers, Samba.

Contact this candidate