Data Engineer with Security Clearance

Positions Description

The Sponsor supports analysts through the provision of large datasets, methodologies, and data visualizations to address pressing intelligence questions. The Sponsor requires support developing and maintaining a cloud-based data environment to transport, store, extract, transform, and load (ETL), and disseminate data solutions. The Sponsor must have an active TS/SCI Full Scope Poly clearance and needs experienced support in data engineering, cloud architecture, and application development. The work includes engaging regularly with data scientists, analysts, and managers. The Data Engineer will assist with strategic planning and oversee implementation of the Sponsor's cloud-based data environment, to include mapping of data sources and access controls. They will develop code, data models, and documentation to Sponsor standards; provide systems administration and programming support for ETL processes and data infrastructure efforts; and train and conduct knowledge transfer to team members on issues and technologies related to Sponsor ETL process, on premise high capacity compute cluster, and administrative duties. The Data Engineer will coordinate with external data and platform providers to ensure the smooth functioning of the Sponsor's systems and data flows, and to accomplish any needed changes and coordinate with experts to assist with technical aspects required to acquire new datasets or data management technologies for inclusion in the Sponsor's environment. They will also support the cross-domain transfer and integration of data.

Required Skills

(Mandatory) Demonstrated experience serving as a technical liaison between system engineers, data engineers, data scientists, analysts, and non-technical managers and personnel.

(Mandatory) Demonstrated experience with AWS cloud services, including long-term storage options, and cloud-based database services such as Databricks or Elastic MapReduce (EMR).

(Mandatory) Demonstrated experience with SQL database structures and mapping between SQL databases.

(Mandatory) Demonstrated experience in large-scale data migration efforts.

(Mandatory) Demonstrated experience with database architecture, performance design methodologies, and system-tuning recommendations. Preference for familiarity with Glue, Hive, and Iceberg or similar

Demonstrated experience with Python, Bash, and Terraform

Demonstrated experience with DevSecOps solutions and tools

Demonstrated experience implementing CI/CD pipelines using industry standard process

Desired Skills (Desired) Demonstrated experience with the Sponsor’s data environment and on-premises compute structure.

Demonstrated experience with Data Quality and Data Governance concepts and experience.

Demonstrated experience maintaining, supporting, and improving the ETL process through the implementation and standardization of data flows with Apache Nifi and other ETL tools.

Demonstrated experience with Apache Spark