Storage Engineer
Resume:
John Holcomb
Chantilly, VA
ac1foj@r.postjobfree.com
SYNOPSIS
A dedicated and highly experienced IT professional with over 25 years of experience in the high-tech industry providing a wide range of services for the commercial and federal sector including network diagnostic, software product design/development, sales support and technical support.
John has proven to be a highly energetic and seasoned technical professional with the demonstrated ability to transform complex technical challenges into workable and cost effective solutions. He is a results driven, self starter and a solid IT technical leader who takes a hands-on approach to his work. John understands the business and corporate objectives, can effectively lead an operational team to meet planned technical milestones on budget and on schedule.
LinkedIn profile: https://www.linkedin.com/in/johnh2/ - most up to date information
Articles:
Dropped packets and buffer size of NICs - https://community.netapp.com/t5/FAS-and-V-Series-Storage-Systems-Discussions/Dropped-packets-and-buffer-size-of-NICs/td-p/6922
NFS disconnect, EXSi 5.x performance problem and setting - https://community.netapp.com/t5/Data-ONTAP-Discussions/NFS-disconnect-EXSi-5-x-performance-problem-and-setting/m-p/96991
Programing languages
Powershell
Bash/Shell
Python (learning now)
PROFESSIONAL EXPERIENCE
HOB Solutions Reston, VA (01/2017 – Present)
Consultant
Customize or write scripts/programs for automation or monitoring of NetApp system(s).
Examine current NetApp configuration and recommend changes.
Examine NetApp system for inefficiency space utilization, make recommendations.
Diagnose performance problems.
Desktop support, answer questions about basic problems such as backups.
Presidio Reston, VA (08/2015 – 11/2016)
Sr. Data Center Engineer/Professional Service Consultant/DevOps
Install and configure NetApp storage systems for customers, including all flash systems.
Assist customers that were migrating from ONTAP 7mode to cDOT (Clustered Data ONTAP), using 7MTT.
Assisted customer with performance problem on the system that turned out to be a network issue. Looking packet capture and analyzing the data.
Install and configure OCUM and OCPM (NetApp OnCommand products).
Work with NetApp team including regional and USA management on upcoming changes and best practices.
Wrote NetApp setup standards to ensure the team was completing a setup the same.
Write program to QA a NetApp cDOT system after installation and check existing system to generate additional revenue. This QA program checks over 200 settings on the NetApp system that were not covered by NetApp config advisor.
Write a program to automate CISCO UCS installation, this program would cut 50% of man hours to complete the initial configuration and ensure the systems are setup consistently. Most of this was completed prior to being laid off.
Blackboard Reston, VA (10/2009 – 05/2015)
Cloud Services Principle Storage Engineer
The company hosts applications for clients in 5 countries with 9 data centers. Storage is used for clients with strict SLA; if uptime is missed uptime clients are owed a credit. Hosted applications include Oracle, MySQL, Microsoft SQL Microsoft Exchange and internal developed applications. All the servers are setup without local hard drives, the systems boot using volumes on the NetApp filers. The company has over 180 NetApp systems, over 40PB of data. Writing BASH/Shell and PowerShell programs, which monitor various parts of the NetApp filers (SNAPMIRROR lag, snapshot schedule, find old snapshots, monitor system changes, locate unused volumes and QTREEs, find disk errors). These programs have helped us automate a lot of our daily tasks and reclaim over 160TB of overhead space. These programs have prevented over a dozen severe outages. Reconfigured snapmirror to cut the transfer time that ranged from 9 to 16 hours to 4 to 6 hours. Redesigned a cluster which resulted in a performance increase by over 66%. Our main location updates over 2100 volumes, about 6TB nightly of changes. Working on OnTap 8.3 cluster mode setup. Finding network problems such as dropped packets. Currently working on Openstack
Design new clusters, set best practices, find performance problems before support or the client is aware.
Manage over 180 filers with SAS/SCSI/SATA/SSD drives, over 40PB of data.
Install (rack, cable, install cards) new filers. Set standards for new clusters, rack location, card location and MPHA cabling. Instruct staff in other countries that have not been trained on NetApp systems to cable them
Write programs to automate routine tasks, such as VSM, QSM, quota and exports file cleanup. These programs have saved the company over 1 million dollars and eliminated the need to purchase two additional clusters. Found over 80TB of space not used and not billed to clients. These programs also located lots of potential problems which would have caused outages (wrong partner statement in RC files, wrong MTU size, many others). These problems have prevented a dozen outages. These programs have found problems that NetApp was not aware of it.
They have two local data centers which use to mirror data between them; the board wanted the mirrors to be at least 1000 miles away. I setup three clusters in a new data center. I wrote a program to move the mirrors to the new data center while keeping the current mirrors running until the new mirrors completed. I successfully mirrored over 1.5 PB of data without incident. I wrote a program to maintain snapmirrors on a daily basis, if the source volume was increased the destination is increased, if the lag exceeds our limits, start an update or send an email about any problems found. This saved operations man hours that use to do this on a daily basis.
Produce DFM/Opsmanager reports on aggregate IOPS and volume IOPS. Wrote programs to use DFM reports instead of issuing filer commands.
Show the benefits of Dedupe and performance impact. This has cut space usage by 50%; one volume is over 90%. Current space savings is over 280TB, saving the company about $3M.
Wrote a program to ensure snapshot schedule based upon our standard setup for various type of volumes. Wrote a program to clean up snapshots older than 6 months.
Installed and configured 10 HA pairs of FAS3270s, six cDOT systems with 6280/6290/8060/8080.
Reconfigured snapmirror to cut back up time from 9-14 hours to 3-5 hours. If they had to purchase hardware compression it would have cost $1.5M. Daily backups for the US are over 2100 volumes, 10TB.
Wrote a program to filter out certain messages out of logs so that operations can review all logs daily. Work with operations staff when a problem needs to be escalated or automate tasks or automate cleanup.
Read perfstat and statit reports to diagnose performance problems.
Consultant Reston, VA (04/2009 – 10/2009)
Help customers with various projects.
NetApp McLean, VA (07/2006 – 04/2009)
Professional Services Engineer level 3 (NCIE+SAN)
Install and configure NetApp equipment for customers covering DC, MD and VA
Work independently, contact customers, consult with customers, keep project information up to date and keep track of expenses.
Working in high pressure situations, a customer had an outage that caused over 1 million customers to be out of service. On call for P1 cases working with hardware issues and network problems.
Performed high profile “head-swaps” as well as large scale installations, changes, and break-fix operations with world-wide visibility.
Performed system upgrades (software and firmware) and head swaps.
Setup SAN (FCP and ISCSI), define LUNs.
Setup NAS (Windows or NFS shares) for users and servers (multiple operating systems).
Setup SNAPMIRROR, SYNCMIRROR and SNAPSHOT.
RDS setup, define volumes, shares, LUNS and document setup (including VISIO drawings).
Work with customers to define power requirements for controller, disk shelf and rack/cabinet. Assist customers with cabinet/rack layout.
Help customer’s setup equipment in large data centers (QWEST, SAVVIS, DuPont Fabros, Equinix and many others).
Develop procedures
Train new staff
Present audits to customers
Generate leads for sales department
Worked with customers and partners to resolve issues fairly and quickly for all parties involved and consistently volunteer for tasks and projects to increase my skills as well as leverage my expertise
Training classes: CISCO MDS Fundamentals, Performance and Tuning, SnapMirror, Metro Cluster.
Independent consultant (Oct 2002 – 07/2006)
NetApp reseller/partner
Completed basic installation and setup for customers in DC, Delaware, Maryland, Pennsylvania and Virginia.
Network and Server Management Consultant (1 year):
Directed technical team (2 in the US and 5 in India), supported the main office and an office in India. This company had Windows 2000 servers with Active Directory, Exchange 2000 servers and SQL 2000 servers. Designed technical support call center (ACD). Setup VOIP with QOS to India (all customer service calls were routed to India).
Setup and maintained firewall and virus protection security
Setup call center for technical support
Setup QOS and VOIP which provided critical daily videoconference, file transfer between Virginia and India.
Developed and implemented processes for server backups, virus protection and security
Performed Microsoft SQL DBA functions
Developed organization, process and technical environment for corporate IT help desk
Provided direct customer interface through the corporate sales force to support customer SQL server plans and implementations
Setup additional Microsoft servers, added them to Active Directory.
Maintain Active Directory, setup new users, remove terminated employees. Normal day-to-day operation of active directory.
Oracle (short term contract)
Oracle was contracted by the Department of Health and Human Services (HHS). HHS wanted to consolidate all financial systems from 13 agencies to a single financial system. The consultants for Oracle work directly with HHS CIO and CTO. Part of this project was to write an RFI that met government standards that would help select a federal data center to host this financial management system. This RFI will also be used select a federal data center selected to be used for a consolidated email server supporting 65,000 users.
Small Business:
Setup and wire new phone system
Setup network and computer protection (firewall and virus protection)
Setup software to become a paperless office
Repair computers and peripherals
Setup disaster recovery
Ocular Networks/Tellabs Reston, VA (Feb 2001 – Oct 2002)
Led a technical team and provided hands-on expert level technical support for daily corporate operations out of Reston Virginia. Provided support for Windows NT servers, Oracle DB, Agile, Rational ClearCase, Microsoft Exchange server and Internet connectivity for internal operations.
Cut corporate long distance cost by 66% and maintained 99% uptime for production systems
Enhanced security with firewall and virus protection
Setup ERP server which was implemented into production in record time to meet key corporate milestones
Setup Rational ClearQuest server with Oracle database.
Setup company Intranet, designed web pages and maintained Extranet
Developed and implemented requirements for new PCs resulting in software build times being reduced from 20 hours to 8.
Reduced IT costs in proportion to increased workload and demand; the company doubled in employees without adding any additional IS/IT staff or costs.
Setup IT help desk and used Clarify to keep track of problems.
Setup VLANs to prevent broadcasts when testing internally developed network products
Successfully integrated Ocular Networks LAN into Tellabs
Best Software Reston, VA (1999-2001)
Provide technical leadership and hands-on day-to-day operation to support a major application software vendor. Provided a Network Operations Center for ASP (Application Service Provider) customers as well as technical support to Microsoft IIS, Windows NT server, IIS and SQL 7. Additionally provided operational backups, review event logs customer system support and troubleshooting.
Developed RFP’s to support selection of hosting providers.
Setup servers for testing and benchmarking.
Hosting provider interface; new server setup, contract services, coordinate outages and problems resolution services.
Enhanced system uptime to exceed 99% for 9 BEST software customers.
Installed VERISIGN Global Site SSL certificate
Wrote stored procedures to automatically perform incremental backups and automatically find databases as they were added.
Wrote stored procedures to perform weekly maintenance, database checks and reindex all tables.
Performed firewall log checking to prevent any potential security breaches.
Reviewed and tested Microsoft patches for potential security problems.
Performed network monitoring using network analyzer (SNIFFER) to ensure traffic was encrypted.
Provided direct interface to Microsoft for timely problem resolution.
Provided Software and operations Sales training on BEST ASP offerings;
Provided Sales and customer interface for BEST IT operations new clients
Setup and managed application product Quality Assurance labs.
Tuned NT and SQL servers for maximum performance and corporate profit
Supported an increase from a max of 70 concurrent server users to over 250.
Wrote requirements for hosted applications and worked with the BEST Product development teams to provide timely fixes for customer problems/technical issues
Developed annual Operations capital budget and projected expenses.
Best Software Reston, VA (1994 - 1999)
Network Services – Provided general expert level technical support for internal BEST Network services operations. Developed specifications and design and provided support to LAN, Windows NT/Novell servers. Provided support of Cisco routers and internet connectivity.
Provided day-to-day operational technical leadership for BEST Software IT organization. Worked successfully with BEST executive management to implement upgrades to phone systems, new servers, LAN/WANs, while simultaneously providing day to day support and maintenance of exiting ACD and PBX phone systems.
Successfully upgraded the entire network from 10 BaseT shared network to a 10/100 BaseT to each desktop with a GIG backbone.
Converted from Internet connection over a WAN to a dedicated Internet connection.
Setup JDE GL application running on Windows NT server with IIS, SQL and Terminal server.
Assisted internal customers with e-mail problems on our Microsoft Exchange and Lotus Notes server.
Upgraded users desktop systems as well as phone system from a key system to a PBX
Negotiated long distance contracts that resulted in cut of BEST Software long distance cost
Designed and delivered CTI (computer telephony integration) on an ASPECT ACD for BEST technical support operations, on time and within budget.
Contact this candidate