Sign in

Systems Engineer

Garland, Texas, United States
January 14, 2018

Contact this candidate

Resume of Chris Vail Page: *

Chris Vail

**** ********** *****

Garland TX 75044

(214) ***-****


Winter 2017-18

Professional Summary

I am a Senior Unix Administrator/Engineer/Architect and Auditor with knowledge of allied disciplines such as, security, storage, networking and physical environment. I fix computers; I make them work. I am comfortable in level 2, 3 and 4 environments. I can plan, build and move entire Data Centers. This is my 35th year as Unix Administrator/Engineer: I bring a lot of maturity with me. I am not merely a technologist, but a good business person. I am friendly and always ready with a joke or a funny story. I offer a successful record of technical leadership.

I am an "OS Polytheist": I do not care which variant I work on. I do some very interesting things with digital media. Unix is not merely a profession, it is an avocation. Systems Security is the only technical religion I have. I am intimate with high-demand, intense environments where human lives and/or trillions of dollars are at stake. Most of my positions have been high-impact & high stress. Usually I am brought in after a disaster or architectural failure to provide a roadmap to future success.

I am expert in Level III/IV Systems Documentation, Auditing, Security, and Architecture. Purchasing: Most technologists do not like this aspect of the job. But I pride myself on writing RFQ’s and beating up salespeople. My record is 70% off list price.

Unlike most technologists, I have good 'people skills'. I am a good teacher and excellent communicator. The Unix joke is that I was 'compiled with the -v flag'.

I have pressed the EPO button and didn’t get fired for it. Technical Skills


P Series: P5, P6, P7 and old RS6000 hardware.

HP9000: PA-RISC & Itanium: N class, A class, Superdome and 8800 series.

Sun Hardware: Ultra and Enterprise, both SPARC and Intel.

Multiple Intel platforms: Dell and HP mostly, but dozens of other brands also. Operating Systems:

Linux: Several variants including CentOS/RedHat through version 7.1, Ubuntu through version 16.04LTS, Scientific, SUSE, Fedora, Debian (on several hardware platforms), Mint, and Tiny.

AIX through version 7.1

HPUX through version 11.23

Virtualization: KVM and VMware

Important skills:

Shell Scripting: Subject Matter Expert (bash, ksh, sh) across hardware & OS platforms.

Ruby, Chef, Perl

Storage: EMC and Hitachi

Networking: anything TCP/IP, including NFS, Samba, DNS, DHCP, SSH and a dozen others. I've been doing networking for 30 years. Am comfortable with Wifi and router hacking. Veritas Netbackup, Tivoli Workload Scheduler (TWS), Tivoli Storage Manager (TWS), Automated Cartridge System Library Software (ACSLS), Altiris Bladelogic,


The University of Texas at Dallas, Richardson, Texas, Spring 1979-Summer 1980, Bachelor of Arts in Drama. The University of North Texas, Denton, Texas, Fall 1976-Spring 1978 Certification/Training

HP Certified IT Professional-Operating Systems: Certificate #00011168942 Specialized training in Unix Security & MC/Service Guard by HP CompTIA Security Plus

SAN Training by EMC

Tivoli Workload Scheduler v7

Resume of Chris Vail Page: 2

Work Experience: Decade 3

Nov 2014-Present: Raytheon, Inc. Senior Hardware and Infrastructure Engineer II. Status: waiting assignment. My activities are considered proprietary, but are consistent with the rest of my career. Received a patent, and a small award recognizing the money I saved the company by my industry knowledge. Was funded 3 times ($65,000) for the purposes of researching, developing and promoting my patent. Built and maintained a 109 node Hadoop cluster, hardware and networking on horrifically out-of-date and slow/broken hardware. Again: I used my shell-script skills often. Primarily these were used for performance tuning and maintenance of the Hadoop (Cloudera, but all FOSS) cluster. That project was moved to SECRET.

August 2013-April 2014: Dell Inc. to Hilton Hotels International. PCI Auditor: Red Hat, Solaris, AIX, and HP OS's, about 300 frames and a large number of virtual devices. Auditing means that system-wide configuration information was collected and compared with PCI-DSS 2007. Where discrepancies were found, then Change Requests were filed and an outage taken for implementation. This was followed by a second configuration examination to confirm that the change took place. The work involved extensive shell scripting (across platforms). I set the standard for shell script publication (form and style). I also provided mentoring and leadership to the rest of the team. This was my 6th PCI audit.

Sep 2012-July 2013: Wipro to Nieman Marcus Group Inc: Senior Unix Administrator. AIX, Solaris, Linux. Provide leadership, vision and standards for the Unix team. This was not a hands-on position but focused on documentation, setting quality standards and best practices. I also taught shell scripting to the whole team. I was able to document an increase of 7 minutes, average response time (against a 30 minute SLA) for P1 outages because of my scripts and procedures. I also assisted the PCI team. A very famous data theft occurred after the audit, but the compromised systems were those that management had taken out of scope for the PCI audit (against the technical recommendation of my team).

May-Aug 2012: Newdata Strategies to JCPenney & Co. SAN/Storage Tivoli Storage Manager (TSM) & ACSLS, Solaris, AIX, Linux (RedHat & SUSE) Administration. Several thousand servers in two remote data centers with Oracle/Sun/StorageTek SLA-8500 storage libraries, EMC DMX SAN. Level II administration, trouble tickets, on-call pager. As an organization, JCPenny is dying, and I was brought in to replace persons who had been laid off after

(sometimes) decades of experience. I was hired as an AIX administrator, but 'shanghaid' to tape administration as they had also fired the entire tape administration team. The TSM installation was very fragile, with constant outages. May-November 2011: Stark Technologies to City of Dallas; San/Unix Administrator. Level III/IV support of Hitachi Storage Area Network & Linux/Solaris (9&10) systems. The hardware was mostly older Sun/Oracle systems, 25 frames with several virtual devices. I also administered several Redhat 5.5 systems on Dell hardware. The SAN consisted of older tape systems (Powderhorn 9310 w/ 9940b) and Hitachi 9980 w/FICON for Amdahl mainframe. Newer Hitachi systems are USP 600, AMS1000 & AMS2300. There were several Dell/EMC devices administered with Navisphere. The fiber switches were McData 6140.

March 2010-April 2011: Interop Technologies, Inc. Night Shift Systems Administration; Network Operations Center. Respond to alarms, break-fix and change tickets. Write documentation and respond to 24x7 crisis situations. Most of the problems were network related. Many trouble tickets involving arcane/obscure knowledge (not in any book) with very few standards, no support for either hardware or software, and very little support for anything else. Only the most senior Engineers have night shift because there is so little support from internal or external resources then. The problems are the most profound because that is when systems are taken down for maintenance. Interop systems process 10’s of millions of SMS messages per hour, and if even one of them does not go through, the night shift Engineers have to deal with it.

Work Experience: Decade 2

Sept-December 2009: CDI Corporation to IBM: Unix Systems Security Analyst. Analyze Unix Systems and Security logs for irregularities. As auditor, my job was to pass judgement on the technical appropriateness of the activities of the systems administrators. This was accomplished by analyzing system logs. The end customer was American Express, the platform was primarily Solaris, but also consulted on AIX and Linux (Red Hat) systems. Sept-Dec 2008 : HLP Solutions to Alliance Data Systems: Unix Engineer/ Administrator; full charge level 2 & 3 systems administrator including AIX through 6.1, Sun Solaris through version 10. Projects included weekly security analysis, shell scripting, patching, Altiris and Blade Logic. All systems were PCI compliant. See also July 2006. These systems were purchased by Heartland Payment systems and were still in use in 2012 when I applied for a position there. Resume of Chris Vail Page: 3

July - Aug 2008: Encore 2 to Levi Strauss & Co., Unix Administrator, Level 2. Full charge administrator for about 60 HP and 10 AIX systems. Volume Manager, networking. Remedy ticketing, 24 hour pager. Level 4,3,2,1 severity support calls. This was a fill-in position for another Administrator who was on FMLA. April-May 2008: Ajilon Consulting. I was hired for 5 weeks 'on the bench' in anticipation of a contract that never was signed. I had no assignment, no duties during this period of time. It amounted to a 5 week, stay-at-home vacation. September 2007 – Feb 2008: Modis, Inc to Fujitsu Network Communications (FNC): Product Support Engineer; final level of problem escalation and resolution. This was not a Systems Administration position, but the software does run on the Solaris Operating System. The primary customers are AT&T, Verizon, and a number of other major Telecom companies. Much time was spent on software engineering, documentation and debugging. March-August 2007; Tek Systems to The Home Depot. Senior Unix Administrator (Level IV). Started as PCI auditor but was shifted to the hardware refresh project. THD had more than 2200 stores, each with at least 6 very out-of-date servers and OS versions. The data centers in Austin and Atlanta (more than 2000 servers) were to be modern- ized to IBM AIX P5/P6 on 5.3/6.1 with LPARS, and HP9000 Itanium with Itanium Virtual Machines, LPAR, NPAR; all with Service Guard. I maintained/updated a LOT of ancient and out-of-date shell scripts to adapt to the new computing paradigm. THD had a famous data theft in 2015 but was predicted in 2007 (I was in the meeting). December 2006-Feb 2007: Insight Global for Hansen, for the San Antonio Water System (SAWS). Title: Technical Resource Lead. I provided high-level architecture and design for a contracted ERP system. Responsibilities: contract variances and escalations, consulting on security. I instituted PVCS control for shell scripts. I mostly wrote documents and procedures.

Aug-Nov 2006: Unix Engineer: The Insource Group to Pepsico Business Systems Group. Level III support for more than 40 Pepsi brands, including Frito-Lay, Quaker Oats, Tropicana and many others. Projects include Disaster Recovery, lots of documentation and shell scripting. Golden Image, patching, day-to-day trouble ticket resolution, mentoring, crash analysis. Mostly support the AIX systems, focusing on the 4.3-->5.3 upgrades, including NIM. Jun/July 2006: IT Auditor with HLP, Inc, to Alliance Data Systems. This was a compliance project for 108 AIX & Solaris Systems. Wrote a library of audit shell scripts. The standard is the PCI or Personal Card Industry (Visa, AMEX, MasterCard). Write security standards, pre- and post-remediation audits. See also Sept-Dec 2008. Nov 2005-April 1, 2006: IT Auditor/Systems Administrator with Matrix Resources Inc. on contract to Aegon Direct Marketing Systems (now purchased by Transamerica) Responsibilities: Sarbanes/Oxley compliance, LPAR/DLPAR, Shell scripting, Secure Shell implementation, Systems Security and production support on Sun/Solaris and IBM/AIX systems. The server mix was 70% AIX, 25% Solaris, and 5% other. December 2004-December 2005: Unix Systems Administrator & Change Coordinator with Argus Connection on contract with ExxonMobil (XOM). Worldwide responsibility for more than 600 Unix systems, mostly HP and IBM with a few Sun. Responsibilities: 24 hour support for global operations, mostly SAP, Oracle and custom/obscure/obsolete software. Data Centers are in Dallas, Houston, and Fairfax VA and several smaller locations worldwide. There just isn't a Unix sub-specialty that is not in use somewhere in this high-security environment. Some of the buzzwords include Power Broker, DCE, NIS, NFS, Samba, MC/Service Guard, HACMP, SMTP, SNMP, EMC/BCV & LVM, NPAR/VPAR/LPAR/DLPAR. Used the Remedy ticketing system, HP Openview OPC/ITO monitoring. Projects included new system commissioning, periodic patch cycle, security, shell scripting, and many others. When Hurricane Rita threatened the Texas coast, XOM closed its Houston Data Center and failed 600+ physical systems and many hundreds of virtual systems to to Dallas. I was given the Disaster Recovery plan and was responsible for the fail-over of what was then the largest company in the world. I didn’t do it all myself, of course, but it was my responsibility to ensure that the plan was implemented appropriately. It was a very long weekend. I authored the RCA document that went to Senior Management. May-September 2004: Storage Administrator with Artech Information Services on contract to IBM Global Services on contract with JP Morgan/Chase in Dallas. The hardware platform was mostly IBM AIX 4.3X, with some 5.1. The storage included several Symmetrix 8830 & 8430 and several DMX series systems, along with mostly McData

(Connectrix) switches. Capacity was ~500TB of disk storage, with ~3PB of tape in 3494 enclosures with LTO-2 drives with Netbackup. I wrote several shell scripts to analyze the configuration of switches and SAN configurations. The storage software included PowerPath, Volume Logix, Enterprise Control Center (5.1) and ESN Manager. Resume of Chris Vail Page: 4

January 2004 – May 2004: Contract Systems Administrator with Aerotech, Inc at Crossroads Systems, Inc.. Responsibilities: Install, configure, and especially debug Unix and Win2k systems in the Interoperability Lab. This included a wide variety of Unix and Win2k systems, including Sun, HP, IBM-AIX, Tru64, IRIX, and other hardware/ software combinations. Also provide senior level (last ditch effort) customer service assistance to the Product Engineering lab. At the point an issue gets to the PE lab, every other technical resource has been exhausted, and a firmware revision needs to be developed and tested. Consulting with them, I helped build out the hardware, OS and application environment so that the customers' issue could be duplicated. The Interop lab was dedicated to testing the interoperability of Crossroads hardware, and firmware with a large cross- section of existing industry standard OS's and hardware configurations. Mostly this has to do with Fiber Channel connectivity. Hardware in the InterOp lab included Adic, ATL, Brocade, Cisco, Clariion, Compaq SureStore, EMC 5830 & 36XX, Fuji, HP, Sun StorEdge, StorageTek (L180), Vixel, Xiotech, and a number of smaller systems. Tape systems include DLT, SDLT, LTO, AIT, 3590, 9840 and a lot of obscure/miscellaneous types. OS's include AIX through 5.1, Solaris through version 9, HP through version 11.22, Linux (Red Hat) through version 0.9, Windows through Advance Server 2003. Connectivity included Fiber Channel (both copper and fiber), Gig Ether and ESCON. January 2001 - October 2003: Systems/SAN Administrator, Electrical Reliability Council of Texas (ERCOT). Responsibilities: Install, configure, and maintain a large IT infrastructure, primarily Oracle (8 & 9) running on HP/UX 11.0 & 11i. Initially 8 servers; L & N class, more than 30 systems, including RP8400 (+NPAR), RP7410, RP5430, V2600. I was intimately involved in the Disaster Recovery and Security/Hardening projects, and wrote many of the scripts and much of the documentation for these. I also handled all issues related to Citrix Metaframe for Unix. I was peripherally involved with the Solaris (v8 & 9 on 220r/420r) team, primarily helping debug really tricky issues and assisting with shell scripting. The SAN included 2 EMC Symmetrix 8830 (59.8TB) with SRDF, a Symmetrix 8430 (17TB), and one Clariion CX600 (80TB with Navisphere), all using PowerPath. ERCOT used Veritas Netbackup v3.4 and later v4.5, and my role was to write the scripts, configure/install and maintain the TSAN. Initially, this system had (12) SDLT tape drives in a StorageTek L700 array (30TB). I was part of the team that migrated it to (20) StorageTek 9940B drives in a PowderHorn 9310 enclosure (1PB). Both SAN and TSAN used EMC Connectrix (OEM Brocade & McData) 64, 32 & 16 port hubs.

Work Experience: Decade 1

March 1994 - January 2001: Enterprise Systems Administrator. Texas Industries, Inc. (TXI), Responsibilities included: System Architect/Administrator responsible for the tactical deployment and maintenance of the enterprise Unix & NT systems. These were several SCO Unix Intel-based systems, 2 RS-6000 AIX RISC-based systems, and 14 Sun Sparc

(several multiprocessor E-series machines, plus a few Ultra machines), Solaris 2.51, 2.6 and 2.7 systems. I was personally responsible to creating and migrating the company to DNS. There were also 25 departmental and enterprise NT servers running mostly on Compaq. I Designed and implemented a 3-tiered environment using 10Base-100 and TCP/IP internetworking. I connected the company to the then-new Internet, created its first website, sent and received the first email. Installed DHCP, NFS, NIS, and inter-server connectivity and scheduling. Installed X terminals, printers, modems, E-mail, gateways, backup procedures and security. Responsible for the maintenance of the power grid, environmental systems and the data center. Was “technical lead” for the entire company, proposing new technologies

(such as imaging and WWW). I taught ksh shell scripting to the development team. My Business/Technical experienced prior to 1994 is summarized in the interest of brevity: A degree in Drama is another term for ‘unemployed’. I drifted into a job at Radio Shack which was then a major retailer of computers. I installed my first Unix system (it was Xenix then) in June 1983. When the store failed in 1987, I went to work for my customers as what we now know as a Systems Administrator, but which job classification was not recognized as a separate discipline until the early 90’s. Initially, I did some application development in Profile/Filepro as well as a lot of things with various OS’s.--including Novell. I provided user training, telephone support and technical writing, and wrote financials and multi-level marketing systems, dispatch systems, inventory control with communication, spreadsheets, mailing (pre-sort) systems, external communications for a daily newspaper, word processing for visually impaired persons, and others. I migrated a lot of clients from Radio Shack hardware to Intel-based systems using SCO. I worked for Texas Instruments and a furniture company. I was senior technical sales repre-sentative for a computer wholesaler. I had a variety of technical and business roles. The details will be supplied upon request. Prior to January 1982, I held positions unrelated to the data processing industry after college. Details will be provided upon request.

Other Interests:

Unix is not merely my profession but also my hobby. My home LAN currently contains 1 of each of the following: Laptop running Ubuntu 14.04, Win 7 Desktop, Fedora Unity 14 Media Server, Sunblade 150 with Solaris 10, HP-UX Resume of Chris Vail Page: 5

workstation with HP-UX 11.11, and AIX 5.3 on a B50 server. I also have a wireless Internet Radio receiver. I am active with the Dallas/Fort Worth Unix Users Group ( I participate in the once-a-month booth at a computer swap meet and help people install Unix, Linux or BSD on their computer systems. I am becoming the go-to guy for Linux-based home media servers in the group. Languages: Advanced Conversational Spanish, Geek

Other Experience

October 1991 - May 1992: Computer literacy instructor, Visions Ministries of Denton, Texas. June 1984 - 2017: I pioneered the Jail Chaplaincy and Homeless outreach programs through my church. 1974, 1976, Volunteer public health worker--vaccinated more than 6000 children in Nicaragua. 2008: Volunteer Balloon Handler in the Dallas Childrens’ Christmas Parade.

Contact this candidate