Aws Python

PROFESSIONAL SUMMARY

Around **+ Years of experience in Devops, Data Platform Engineering,Infrastructure engineer and Data Operations with knowledge Bigdata,Pyspark, AWS, GCP, CI/CD, EMR, MapR, RDS, CloudFormation and Lambda

Developed and deployed AWS resources with CloudFormation and Python. Worked as data platform engineering. Worked on AWS migration.

Deployed Hadoop/Bigdata platform on AWS and also on prem. Did the migration from on-prem to AWS,

Worked on Pyspark programs for ETL. And for data analysis on MapR, AWS-EMR, AWS-Glue and on Hadoop clusters.

Monitoring the resources and applications.

Worked on terraform.

AWS lambda functions development using Python which involves me 5 migration projects which also include QuickBase application beside Hadoop migration projects.

Worked as Data Ops Engineer by leveraging my Hadoop admin skills by picking up the python with spark.

Apache Spark Data Frames/RDD's were used to apply business transformations and utilized Hive Context objects to perform read/write operations.

Built the framework with python to run application from spark, Hive, Hadoop, and AWS.

Converted pipelines from hive to spark.

Hadoop administration with automation.

Migrated the applications from on-prem to cloud.

Hands on experience to install and configuring Hadoop Clusters (MapR, Cloudera and Hortonworks), AWS EMR clusters.

Education: Bachelors in Electronics and communications from JNTU, Hyderabad, India – 2007.

Certifications:

Microsoft Certified Professional in 2014

MapR Hadoop Admin 2016

AWS Architect associate and SysOps

AWS Devops professional

Google cloud admin 2020

Technical Skills Summary

Hadoop Eco-Systems Hive, Pig, Flume, Oozie, Sqoop, Spark, Impala and HBase

Operating systems: RedHat Linux 5.X, 6.X, Windows 95, 98, NT, 2000, Windows Vista, 7

Configuration Management Tools Puppet

Database Oracle (SQL) 10g, MYSQL, SQL SERVER 2008

Hadoop Configuration Management Cloudera Manager, Ambari

Monitoring Tools Ganglia, Nagios

Scripting Languages: Shell scripting, PowerShell.

Configuration / Protocol: DNS, DHCP, WINS, VPN, TCP/IP, SNMP, IMAP, POP3, SMTP, PKI, DFS

Ticketing Systems: Remedy, Service Now, IBM Tivoli

Backup software’s: Net-Backup, Tivoli, Com vault, NT Backup, DPM 2012

Professional Experience:

Intuit, Mountain View, California (June 21st, 2019 – Present)

AWS resources deployment with development with CFT and Python along with CI/CD.

Monitoring the resources and applications.

Google cloud platform framework implementation and development.

Data engineering of internal data with EMR, S3, RDS, EC2, EFS and load balancer.

AWS development to create and maintain all resources.

Design of pipelines, update documentation and implementing.

Create CloudFormation, python scripts for AWS resources.

AWS migration of Quick base,Slack,Box,AgilePoint, Hadoop cluster and other application.

Automation carrying over with workato, ServiceNow.

Python development for REST API, PYSPARK and automation with GIT to achieve CI/CD.

Writing Box/gdrive related Java code and deploying with Kubernetes.

Implemented secure cloud architecture based on AWS to make sure applications are reliable, scalable, and universally available. Built and configured a virtual data centre in the Amazon Web Services cloud to support Enterprise Data Warehouse hosting including Virtual Private Cloud (VPC), Public and Private Subnets, Security Groups, Route Tables, Elastic Load Balancer.

Good experience with cloud migration of application to Cloud Amazon Web Services

Implemented Amazon EC2 by setting up instances, Virtual Private Cloud (VPCs), and security groups and by automating the process of launching and stopping/terminating EC2 instances on AWS with BOTO3.

Developed CloudFormation scripts to automate EC2 instances and managed 200+ AWS instances and configured Elastic IP & Elastic Storage in open enrollment period and implemented security groups and Network ACL’s.

Next Gen Technologies Inc

Intuit, Mountain View, California (Jan 2019 to June 14th 2019)

Write CloudFormation scripts and python programs to provision different AWS resources. And for automation.

Monitoring the resources and applications.

Complete spark and hive based ETL framework with Python Pyspark along with AWS services and Cloud infrastructure automation.

Write SQL and NoSQL queries for different data requirements.

Participate in designing and develop pipelines from different type resources.

Deploy AWS EMR cluster and create data for reports using Spark and Hive.

Write CloudFormation scripts to provision different AWS resources.

Manage the AWS EMR clusters to complete data generation.

Analyze data and create data pipeline for data modeling and data quality.

Contribute and implement the continuous integration and continuous delivery pipeline

Ensure data availability, data quality and data modeling smooth process.

Developed and supported software using Python, Bash, MapR, Hadoop, and Amazon Web Services (including S3, EC2, VPCs,EMR,Athena,Glue and IAM).

Act to find out the root cause of any issues, while generating the data using spark and Hadoop eco system components with AWS services and cloud services.

Implement data management using Redshift, HBase, Kafka, spark, hive, kinesis, and Sqoop.

Create reports using AWS Glue, AWS Athena, Hadoop, Spark, AWS EMR, Hive and Databricks Platform.

Comcast, Sunnyvale, California (Jan 2017 Jan 2019- Full Time

Deploy and maintained MapR Hadoop cluster using CI/CD pipeline Chef and Jenkins.

Monitoring the resources and applications.

Developed Python Pyspark programming for ETL, Big data applications, Data pipelines and for Data analysis.

Implemented automation using Python for regular tasks.

Implemented automation for pivot table from AWS Athena table by using Python Pandas and AWS services in Boto3.

Created the AWS EMR clusters using Shell script and Boto3 to run the ETL jobs.

Provisioned the AWS resources using CloudFormation and CI/CD pipelines.

Completed the Data validation using Python framework and AWS services like Athena, Glue and SQL queries.

Developed and supported software using Python, Bash, MapR, Hadoop, and Amazon Web Services (including S3, EC2, VPCs,EMR,Athena,Glue and IAM).

Complete spark and hive based ETL framework with Python/Java/Scala along with AWS services and Cloud.

Write the Python Pyspark code for data analysis and data management and spark development for reports.

Write SQL and NoSQL queries for different data requirements.

Completed the automation to get data and responses using RESTFUL API’s.

DataQuest (Comcast)

Period: March 2016 – Jan 17

Provisioned the AWS resources using CloudFormation and CI/CD pipelines.

Monitoring the resources and applications.

Python ETL with Spark, Hive and building Data pipelines.

Systems administration within Linux OS.

Developed and supported software using Python, Bash, MapR, Hadoop, and Amazon Web Services (including S3, EC2, VPCs, and IAM).

Integrated CI/CD software automation using Jenkins.

Automated and orchestrated tasks using Ansible.

Automated software environments using Chef and developed and supported software using Java.

Additionally, he designed, installed, tested, implemented, and supported a Hadoop clusters.

Installed a Linux operating system and set up Hadoop clusters.

Wrote scripts in Shell and Python to automate system tasks.

Maintained MapR and other versions of big data Hadoop clusters within an operating system.

Performed work automation and job processing using Shell scripts and tools.

Microsoft, Seattle and KPN (Mindtree-Bangalore)

Period: Sept 13 – Jan 16

Position: Senior Engineer

Worked on Infrastructure with automation provisioning

Monitoring the resources and applications.

Systems administration within Linux OS.

Developed and supported software using Python, Bash, MapR, Hadoop, and Amazon Web Services (including S3, EC2, VPCs, and IAM).

Integrated CI/CD software automation using Jenkins

Automated and orchestrated tasks using Ansible

Automated software environments using Chef and developed and supported software using Java.

Additionally, he developed and supported hardware from Dell, HP, and IBM.

Installed and maintained Linux servers and Windows servers, including 2003, 2008, and 2012 AD(Active Directory)

Performed and maintained the Windows 2003/2008/2012 Active directory group policies and DFS managing.

Client/Organization: FRHI, San Jose (ITC infotech)

Period: Oct’12 – Sep ‘13

Position:- IT consultant

Automation with Python, Shell script and Power shell.

Monitoring the resources and applications.

Systems administration within Linux OS.

Integrated CI/CD software automation using Jenkins; and automated and orchestrated task’s using Ansible. Additionally, he implemented nodes on a CDH3 Hadoop cluster on Red Hat Linux;

Imported data from Linux file systems to Hadoop Distributed File System (HDFS). Supported Linux servers and Hadoop clusters; loaded data from UNIX to HDFS;

Supported Windows Server 2003/2008/2012 Active Directory, Exchange Management Console for 2003, 2007, and 2010; monitored trouble ticket queues; Automation with Python, Shell script and Power shell.

set up a domain and active directory on Windows 2008 server.

Monitoring the systems and networks using hp-sitescope, HP NNM, sclar winds and taken appropriate action.

Worked on Symantec backup for the daily jobs and escalated issues.

Monitoring servers for availability, Disk space, Services, LDAP and Port g080 monitoring en AD servers.

Troubleshot WDS/RIS servers related issue.

EDUCATION SUMMARY

Bachelors in Electronics and communications from JNTU, Hyderabad, India - 2007

Contact this candidate