Sign in

Sr. Splunk Engineer, Assistant Vice President - RT

Absolute Opportunities
Monterey Park, California, 91754, United States
November 14, 2018


Sr. Splunk Engineer, Assistant Vice President - RT

Monterey Park, California 91754


Our group operates Splunk, an operational big data intelligence software, as a service for various teams. We are seeking a motivated engineer to continue to build up our Splunk implementation, and to help our customers to fully utilize the power of Splunk.

As a member of the team, the Splunk engineer will implement various solutions on Splunk, as well as supporting test and production Splunk installations.

Successful candidates need to possess expert level hands-on solution building and administrative experience in Splunk.

In additon, this level has all the tasks of a Senior Specialist with the added task of integrating information and network security CSOC and APP-SOC-MONITORING solutions. Enforce Splunk security strategies and support existing Splunk systems in accordance with policies, standards, guidelines and procedures.

This is an expert level position. The expert will possess a level of knowledge and a skill set that goes beyond that of a Sr. Analyst. The expert will have detailed knowledge of several platforms and security systems.


Deployment and support of the full lifecycle of Splunk Enterprise

Design, implement, document, and handle all aspects of Linux as it relates to Splunk

Identify repetitive, manual tasks and automate them

Develop effective tooling, alerts, and response to both identify and address reliability risks

Write and maintain moderately complicated scripts in shell scripting (Bash, Python, Ruby, JavaScript, and/or Perl) in helping to automate and scale

Provide technical leadership and mentor junior team members

Build with quality and integrity


4-5 year minimum experience with Splunk.

5+ years Unix/Linux system administration experience

DevOps experience with Ansible (or Puppet/Chef/Salt), Git and Jenkins

Familiarity with other big data technologies such as Hadoop, Cassandra.

Familiar with one or more languages such as Python, Java, C/C++, shell scripting/regular expression, SQL and HTML/Javascript

A strong understanding of high-traffic, large-scale distributed systems and the ability to perform root cause analysis on stability and performance related events in such environments

Familiarity with continuous integration and continuous deployment systems and the ability to describe pros, cons, and pitfalls of the various solutions.

High familiarity with Git and version control systems

Experience with Linux systems; must understand how processes, users, groups, privileges and package managers work

Hands on experience in backup and restore tools.

Experience with automation and configuration management systems such as Puppet, Ansible, Salt, etc.

Expert proficiency in UNIX scripting languages (Bash, Ruby, Python) and some experience with compiled languages (Go, Java, etc)

Experience with configuration and troubleshooting of Linux, Java, Tomcat, and other middleware technologies

Passion for resolving reliability issues and identify strategies to mitigate going forward

Experience with Cloud Computing platforms (particularly AWS) a plus

Strong Linux system-level analysis capabilities

Passion for clear communication, especially prioritizing concerns to align with the team and business goals.

Deep network analysis experience

Thorough understanding of networking

Support large-scale deployments with data feeds from multiple data centers

Develop Splunk correlation searches to identify and address emerging security threats through the use of continuous monitoring, alerting and analytics

Installing, configuring and administering Splunk Enterprise Server and Splunk Universal/heavy forwarders in large distributed environment

Installing and configuring Splunk apps in a clustered environment

Administering Splunk knowledge objects

Creating roles and user authentication

Integrating events from non-traditional log services

Administering Splunk cluster components (search head cluster, indexer cluster and distributed management console) including version upgrades, permissions, and audit compliance

Mentoring other Information Security team members to support and assist in Splunk-related activities

Assists in setting business driven SLAs and owns evolving the environment to meet or exceed those SLAs.

Performs advanced troubleshooting and issue resolution for all supported systems.

Utilize monitoring tools for performance monitoring and capacity management. Plan proactive system changes/upgrades based on performance and capacity data.

Create and maintain documentation for team standards, procedures, common issue resolution for other IT staff and systems users.

Participate in team on-call rotation schedules. On-call provides 24/7 availability during rotation to support issues and assist team with scheduled operational tasks after production hours.

Position requires working after normal business hours to implement changes to supported systems.

Demonstrate good judgment by escalating issues to the manager when appropriate.

History working with or supporting Microsoft C# applications, SQL Server, IIS, AWS and ntiered solutions

Some experience with T-SQL skills, review T-SQL code and procedures



Required Knowledge, Skills, and Abilities: (Submission Summary)

1. Bachelor's Degree? (required)

2. 4-5 year minimum experience with Splunk? REQUIRED - do not submit candidates that do not meet this requirement.

3. 5+ years Unix/Linux system administration experience?

4. DevOps experience with Ansible (or Puppet/Chef/Salt), Git and Jenkins? List details below:

5. Familiarity with other big data technologies such as Hadoop, Cassandra?

6. Familiar with one or more languages such as Python, Java, C/C++, shell scripting/regular expression, SQL and HTML/Javascript? LIST DETAILS BELOW:

7. Salary Expectation?

8. Must be a US Citizen or Green Card holder?

9. Complete Current Address