Senior Unix/Linux Systems Administrator
Resume:
Chris Vail
Suphur Springs, TX 75482
adb42y@r.postjobfree.com
-
Winter/Spring 2020
Professional Summary
I am a Senior Unix Administrator/Engineer/Architect and Auditor with knowledge of allied disciplines such as, security, storage, networking and physical environment. I fix computers; I make them work. I am comfortable in level 2, 3 and 4 environments where I can plan, build and move entire Data Centers. I have moved Data Centers 3 times
This is my 37th year as Unix Administrator/Engineer: I bring a lot of maturity with me. I am not merely a technologist, but a good business person. I offer a successful record of technical leadership.
I am an "OS Polytheist": I do not care which variant I work on. I do some very interesting things with digital media.
Unix /Linux is not merely merely a profession, it is an avocation.
Systems Security is the only technical religion I have.
I am intimate with high-demand, intense environments where human lives and/or trillions of dollars are at stake.
Most of my positions have been high-impact & high stress. I have never had a failed project, but have had delayed (or redefined) success.
I am expert in Level III/IV Systems Documentation, Scripting Auditing, Security, and Architecture.
Purchasing: Most technologists do not like this aspect of the job. But I pride myself on writing RFQ’s and beating up salespeople. I get a lot more than lunch out of salespeople. My background is in sales: I know how they think.
Unlike most technologists, I have good 'people skills'. I am a good teacher and excellent communicator. The Unix joke is that I was 'compiled with the -v flag'.
I have pressed the EPO button and didn’t get fired for it.
The last page of this document is more important than this page.
Technical Skills
Hardware:
P Series: P5, P6, P7, P8, P9 and old RS6000 hardware.
HP9000: PA-RISC & Itanium: N class, A class, Superdome and 8800 series.
Sun Hardware: Ultra and Enterprise, both SPARC and Intel.
Multiple Intel platforms: Dell and HP mostly, but dozens of other brands also.
Operating Systems:
Linux: Several variants including CentOS/RedHat through version 7.4, Ubuntu through version 18.04LTS, Scientific, SUSE, Fedora, Debian (on several hardware platforms), Mint, and Tiny.
Solaris through v11
AIX through version 7.2
HPUX through version 11.23
Virtualization: KVM and Vmware, VirtualBox
Important skills:
Shell Scripting: Subject Matter Expert (bash, ksh, sh) across hardware & OS platforms.
PCI compliance. The PCI standards include Tripwire by name.
Ruby, Chef, Perl
Storage: EMC and Hitachi
Networking: anything TCP/IP, including NFS, Samba, DNS, DHCP, SSH and a dozen others. I've been doing networking for 30 years. Am comfortable with Wifi and router hacking.
Centrify, Veritas Netbackup, Tivoli Workload Scheduler (TWS), Tivoli Storage Manager (TSM), Automated Cartridge System Library Software (A
Education
The University of Texas at Dallas, Richardson, Texas, Spring 1979-Summer 1980, Bachelor of Arts in Theater.
The University of North Texas, Denton, Texas, Fall 1976-Spring 1978
Certification/Training
HP Certified IT Professional-Operating Systems: Certificate #000********
Specialized training in Unix Security & MC/Service Guard by HP
CompTIA Security Plus
SAN Training by EMC
Work Experience:
June-July 2019: Experis, Inc. to IBM Softlayer. Write and implement Standard Operating Procedures in response to security scans. An example might be to re-mediate troubles in old versions of Red Hat Linux and Solaris Openssl installations.
January-April: Matrix, Inc. to Neiman-Marcus Group, Inc. Senior Unix Engineer III. This is a recall engagement, see 2012-2013. Since then, N-M decided to outsource its entire IT infrastructure. I was brought in to implement Centrify, a third-party Identity Management (IDM/IAM) product that subsumes Unix/Linux identity management into Active Directory. I was a Security Project Team member rather than Operations or Development.
Sept-Dec 2018: Bartech, Inc. to Verizon Wireless. Sr. Operations Engineer. Monitor and maintain a massive infrastructure including RHEL, and Windows 2008, 2012, 2016 using Nagios and Splunk. Change Management, documentation, vendor coordination and implementation, and report after report after report. These are real-time systems that are the Internet. These are single-application, database and ESX Cloud environments.
Nov 2014-Sept 2018: Raytheon, Inc. Senior Hardware and Infrastructure Engineer II. Received a patent in Space technology (I am, literally, a Rocket Scientist), and a small award recognizing the money I saved the company by my industry knowledge. Was funded 3 times for the purposes of researching, developing and promoting my patent (Principle Investigator was the job title at that point).
I built and maintained a 109 node Hadoop cluster running on CentOS 6 & 7. The hardware and networking were horrifically out-of-date and slow or broken. Again: I used my shell-script skills often. Primarily these were used for performance tuning and maintenance of the Hadoop (Cloudera, but all FOSS) cluster. Used DevOps techniques and Chef/Ruby.
August 2013-April 2014: Dell Inc. to Hilton Hotels International. PCI Auditor: Red Hat, Solaris, AIX, and HP OS's, about 300 frames and a large number of virtual devices. Auditing means that system-wide configuration information was collected and compared with PCI-DSS 2007. Where discrepancies were found, then Change Requests were filed and an outage taken for implementation. This was followed by a second configuration examination to confirm that the change took place. The work involved extensive shell scripting (across platforms). I set the standard for shell script publication (form and style). I also provided mentoring and leadership to the rest of the team. This was my 6th PCI audit.
Sep 2012-July 2013: Wipro to Nieman Marcus Group Inc: Senior Unix Administrator. AIX, Solaris, Linux. Provide leadership, vision and standards for the Unix team. This was not a hands-on position but focused on documentation, setting quality standards and best practices. I also taught shell scripting to the whole team. I was able to document an increase of 7 minutes, average response time (against a 30 minute SLA) for P1 outages because of my scripts and procedures. I also assisted the PCI team.
May-Aug 2012: Newdata Strategies to JCPenney & Co. SAN/Storage Tivoli Storage Manager (TSM) & ACSLS, Solaris, AIX, Linux (RedHat & SUSE) Administration. Several thousand servers in two remote data centers with Oracle/Sun/StorageTek SLA-8500 storage libraries, EMC DMX SAN. Level II administration, trouble tickets, on-call pager.
May-November 2011: Stark Technologies to City of Dallas; SAN/Unix Administrator. Level III/IV support of Hitachi Storage Area Network & Linux/Solaris (9&10) systems. The hardware was mostly older Sun/Oracle systems, 25 frames with several virtual devices. I also administered several Redhat 5.5 systems on Dell hardware. The SAN consisted of older tape systems (Powderhorn 9310 w/ 9940b) and Hitachi 9980 w/FICON for the Amdahl mainframe. Newer Hitachi systems were USP 600, AMS1000 & AMS2300. There were several Dell/EMC devices administered with Navisphere. The fiber switches were McData 6140.
March 2010-April 2011: Interop Technologies, Inc. Night Shift Systems Administration; Network Operations Center. Respond to alarms, break-fix and change tickets. Write documentation and respond to 24x7 crisis situations. Most of the problems were network related. Many trouble tickets involving arcane/obscure knowledge (not in any book) with very few standards, no support for either hardware or software, and very little support for anything else. Only the most senior Engineers have night shift because there is so little support from internal or external resources then. The problems are the most profound because that is when systems are taken down for maintenance. Interop systems process 10’s of millions of SMS messages per hour, and if even one of them does not go through, the night shift Engineers have to deal with it.
Sept-December 2009: CDI Corporation to IBM: Unix Systems Security Analyst. Analyze Unix Systems and Security logs for irregularities. As auditor, my job was to pass judgment on the technical appropriateness of the activities of the systems administrators. This was accomplished by analyzing system logs. The end customer was American Express, the platform was primarily Solaris, but also consulted on AIX and Linux (Red Hat) systems.
Sept-Dec 2008 : HLP Solutions to Alliance Data Systems: Unix Engineer/ Administrator; full charge level 2 & 3 systems administrator including AIX through 6.1, Sun Solaris through version 10. Projects included weekly security analysis, shell scripting, patching, Altiris Blade Logic. All systems were PCI compliant. See also July 2006. These systems and procedures were purchased by Heartland Payment Systems and were still in use in 2012 when I applied for a position there.
July - Aug 2008: Encore 2 to Levi Strauss & Co., Unix Administrator, Level 2. Full charge administrator for about 60 HP and 10 AIX systems. Volume Manager, networking. Remedy ticketing, 24 hour pager. Level 4,3,2,1 severity support calls. This was a fill-in position for another Administrator who was on FMLA.
April-May 2008: Ajilon Consulting. I was hired for 5 weeks 'on the bench' in anticipation of a contract that never was signed. I had no assignment, no duties during this period of time. It amounted to a 5 week, stay-at-home vacation.
September 2007 – Feb 2008: Modis, Inc to Fujitsu Network Communications (FNC): Product Support Engineer; final level of problem escalation and resolution. This was not a Systems Administration position, but the software does run on the Solaris Operating System. The primary customers are AT&T, Verizon, and a number of other major Telecom companies. Much time was spent on software engineering, documentation and debugging.
March-August 2007; Tek Systems, Inc. to The Home Depot. Senior Unix Administrator (Level IV). Started as PCI auditor but was shifted to the hardware refresh project. THD had more than 2200 stores, each with at least 6 very out-of-date servers and OS versions. The data centers in Austin and Atlanta (more than 2000 servers) were to be modernized to IBM AIX P5/P6 on 5.3/6.1 with LPARS, and HP9000 Itanium with Itanium Virtual Machines, LPAR, NPAR; all with Service Guard. I maintained/updated a LOT of ancient and out-of-date shell scripts to adapt to the new computing paradigm.
December 2006-Feb 2007: Insight Global for Hansen, for the San Antonio Water System (SAWS). Title: Technical Resource Lead. I provided high-level architecture and design for a contracted ERP system. Responsibilities: contract variances and escalations, consulting on security. I instituted PVCS control for shell scripts. I mostly wrote documents and procedures.
Aug-Nov 2006: The Insource Group to Pepsico Business Systems Group. Unix Engineer. Level III support for more than 40 Pepsi brands, including Frito-Lay, Quaker Oats, Tropicana and many others. Projects include Disaster Recovery, lots of documentation and shell scripting. Golden Image, patching, day-to-day trouble ticket resolution, mentoring, RCA crash analysis. Mostly support the AIX systems, focusing on the 4.3-->5.3 upgrades, including NIM.
Jun/July 2006: IT Auditor with HLP, Inc, to Alliance Data Systems. This was a compliance project for 108 AIX & Solaris Systems. Wrote a library of audit shell scripts. The standard is the PCI or Personal Card Industry (Visa, AMEX, MasterCard). Write security standards, pre- and post-remediation audits. See also Sept-Dec 2008.
Nov 2005-April 1, 2006: IT Auditor/Systems Administrator with Matrix Resources Inc. to Aegon Direct Marketing Systems (now purchased by Transamerica) Responsibilities: Sarbanes/Oxley compliance, LPAR/DLPAR, Shell scripting, Secure Shell implementation, Systems Security and production support on Sun/Solaris and IBM/AIX systems. The server mix was 70% AIX, 25% Solaris, and 5% other.
December 2004-December 2005: Unix Systems Administrator & Change Coordinator with Argus Connection on contract with ExxonMobil (XOM). Worldwide responsibility for more than 600 Unix systems, mostly HP and IBM with a few Sun. Responsibilities: 24 hour support for global operations, mostly SAP, Oracle and custom/obscure/obsolete software. Data Centers are in Dallas, Houston, and Fairfax VA and several smaller locations worldwide. There just isn't a Unix sub-specialty that is not in use somewhere in this high-security environment. Some of the buzzwords include Power Broker, DCE, NIS, NFS, Samba, MC/Service Guard, HACMP, SMTP, SNMP, EMC/BCV & LVM, NPAR/VPAR/LPAR/DLPAR. Used the Remedy ticketing system, HP Openview OPC/ITO monitoring. Projects included new system commissioning, periodic patch cycle, security, shell scripting, and many others.
When Hurricane Rita threatened the Texas coast, XOM closed its Houston Data Center and failed 600+ physical systems and many hundreds of virtual systems to to Dallas. I was given the Disaster Recovery plan and was responsible for executing it: the fail-over of what was then the largest company in the world. I didn’t do it all myself, of course, but it was my responsibility to ensure that the plan was implemented appropriately. It was a very long weekend. I authored the RCA document that went to Senior Management.
May-September 2004: Storage Administrator with Artech Information Services on contract to IBM Global Services on contract with JP Morgan/Chase in Dallas. The hardware platform was mostly IBM AIX 4.3X, with some 5.1. The storage included several Symmetrix 8830 & 8430 and several DMX series systems, along with mostly McData (Connectrix) switches. Capacity was ~500TB of disk storage, with ~3PB of tape in 3494 enclosures with LTO-2 drives with Netbackup. I wrote several shell scripts to analyze the configuration of switches and SAN configurations. The storage software included PowerPath, Volume Logix, Enterprise Control Center (5.1) and ESN Manager.
January 2004 – May 2004: Systems Administrator with Aerotech, Inc at Crossroads Systems, Inc.. Responsibilities: Install, configure, and especially debug Unix and Win2k systems in the Interoperability Lab. This included a wide variety of Unix and Win2k systems, including Sun, HP, IBM-AIX, Tru64, IRIX, and other hardware/ software combinations. Also provide senior level (last ditch effort) customer service assistance to the Product Engineering lab. At the point an issue gets to the PE lab, every other technical resource has been exhausted, and a firmware revision needs to be developed and tested. Consulting with them, I helped build out the hardware, OS and application environment so that the customers' issue could be duplicated.
The Interop lab was dedicated to testing the interoperability of Crossroads hardware, and firmware with a large cross-section of existing industry standard OS's and hardware configurations. Mostly this has to do with Fiber Channel connectivity. Hardware in the InterOp lab included Adic, ATL, Brocade, Cisco, Clariion, Compaq SureStore, EMC 5830 & 36XX, Fuji, HP, Sun StorEdge, StorageTek (L180), Vixel, Xiotech, and a number of smaller systems. Tape systems include DLT, SDLT, LTO, AIT, 3590, 9840 and a lot of obscure/miscellaneous types. OS's include AIX through 5.1, Solaris through version 9, HP through version 11.22, Linux (Red Hat) through version 0.9, Windows through Advance Server 2003. Connectivity included Fiber Channel (both copper and fiber), Gig Ether and ESCON.
January 2001 - October 2003: Systems/SAN Administrator, Electrical Reliability Council of Texas (ERCOT). Responsibilities: Install, configure, and maintain a large IT infrastructure, primarily Oracle (8 & 9) running on HP/UX 11.0 & 11i. Initially 8 servers; L & N class, more than 30 systems, including RP8400 (+NPAR), RP7410, RP5430, V2600. I was intimately involved in the Disaster Recovery and Security/Hardening projects, and wrote many of the scripts and much of the documentation for these. I also handled all issues related to Citrix Metaframe for Unix. I was peripherally involved with the Solaris (v8 & 9 on 220r/420r) team, primarily helping debug really tricky issues and assisting with shell scripting. The SAN included 2 EMC Symmetrix 8830 (59.8TB) with SRDF, a Symmetrix 8430 (17TB), and one Clariion CX600 (80TB with Navisphere), all using PowerPath. ERCOT used Veritas Netbackup v3.4 and later v4.5, and my role was to write the scripts, configure/install and maintain the TSAN. Initially, this system had (12) SDLT tape drives in a StorageTek L700 array (30TB). I was part of the team that migrated it to (20) StorageTek 9940B drives in a PowderHorn 9310 enclosure (2PB). Both SAN and TSAN used EMC Connectrix (OEM Brocade & McData) 64, 32 & 16 port hubs.
On 9/11/2001 I was in the NOC (I was Architect, not Operations) and like everyone else, watched on TV the events unfolding in NYC and Washington DC. I led both NOCs into forming a “Committee of the Whole” and put ERCOT on a war time footing. I was the most senior company officer onsite at that time. We had no established procedures for a war, so we wrote one in real time. Nobody in Texas lost their lights because the systems I designed there were resilient, and the leadership I provided was calm and professional. In the moments after the 2nd airplane hit the 2nd building, I committed the Company to spending $12,000,000.00 per hour to spin all the electrical generators in the State. That decision (and many others) were affirmed unanimously by the Board of Directors the next day, and later affirmed by the Governor and Legislature of the State of Texas.
March 1994 - January 2001: Enterprise Systems Administrator. Texas Industries, Inc. (TXI) Now Martin-Marietta, Responsibilities included: System Architect/Administrator responsible for the deployment and maintenance of the enterprise Unix & NT systems. These were several SCO Unix Intel-based systems, 2 RS-6000 AIX RISC-based systems, and 14 Sun Sparc (several multiprocessor E-series machines, plus a few Ultra machines), Solaris 2.51, 2.6 and 2.7 systems. I was responsible to creating and migrating the company to DNS. There were also 25 departmental and enterprise NT servers running mostly on Compaq. I Designed and implemented a 3-tiered environment using 10Base-100 and TCP/IP internetworking. I connected the company to the then-new Internet, created its first website, sent and received its first emails. Installed DHCP, NFS, NIS, and inter-server connectivity and scheduling. Installed X terminals, printers, modems, E-mail, gateways, backup procedures and security. Responsible for the maintenance of the power grid, environmental systems and the data center. Was “technical lead” for the entire company, proposing new technologies (such as imaging and WWW). I taught ksh shell scripting to the development team.
My Business/Technical experienced prior to 1994 is summarized in the interest of brevity:
A Bachelor’s degree in Drama is another term for ‘unemployed’. I drifted into a job at Radio Shack which was then a major retailer of computers. I installed my first Unix system (it was Xenix then) in June 1983. When the store failed in 1987, I went to work for my customers as what we now know as a Systems Administrator, but which job classification was not recognized as a separate discipline until the early 90’s. Initially, I did some application development in Profile/Filepro as well as a lot of things with various OS’s.--including Novell. I provided user training, telephone support and technical writing, and wrote financials and multi-level marketing systems, dispatch systems, inventory control with communication, spreadsheets, mailing (pre-sort) systems, external communications for a daily newspaper, word processing for a visually impaired person, and others. I migrated a lot of clients from Radio Shack hardware to Intel-based systems using SCO. I worked for Texas Instruments and a furniture company. I was senior technical sales representative for a computer wholesaler. I had a variety of technical and business roles. The details will be supplied upon request.
Prior to January 1982, I held positions unrelated to the data processing industry after college. Details will be provided upon request.
Other Interests:
Unix is not merely my profession but also my hobby. My home LAN currently contains 1 of each of the following: Laptop running Ubuntu 16.04, Win 7 Desktop, Fedora Unity 14 Media Server, Sunblade 150 with Solaris 10, HP-UX workstation with HP-UX 11.11, and AIX 5.3 on a B50 server. I also have a wireless Internet Radio receiver. I am active with the Dallas/Fort Worth Unix Users Group (http://www.dfwuug.org). I am the go-to guy for Linux-based home media servers in the group, and shell-script guru.
Languages: Advanced Conversational Spanish, and Incomprehensibly Dense Geek
Other Experience
October 1991 - May 1992: Computer literacy instructor, Visions Ministries of Denton, Texas.
June 1984 - 2017: I pioneered the Jail Chaplaincy and Homeless outreach programs through my church.
1974, 1976, Volunteer public health worker--vaccinated more than 6000 children in Nicaragua.
2008: Volunteer Balloon Handler in the Dallas Childrens’ Christmas Parade.
2017: Volunteer at Career Day promoting STEM education for women at the University of Texas at Dallas.
Next Page: In August 2019, a recruiter asked me to take a Linux technical test (RHEL 5). It was unusually difficult, as many of the questions had more than one correct answer, and I had to choose them all. The test was ‘smart’, the answers to subsequent questions were always more obscure than the initial questions were . The results of this test are on the next page. This test illustrates why I do not need industry certifications:.
I took the same test in March 2018. My score was lower then, but still very good.
x
Contact this candidate