Sr. Data Engineer

Job Description

Title : Sr. Data Engineer

Location: Oregon

Duration: 12 Months+

Roles & Responsibilities

Perform data analysis according to business needs

Translate functional business requirements into high-level and low-level technical designs

Design and implement distributed data processing pipelines using Apache Spark, Apache Hive, Python, and other tools and languages prevalent in a modern analytics platform

Create and schedule workflows using Apache Airflow or similar job orchestration tooling

Build utilities, functions, and frameworks to better enable high-volume data processing

Define and build data acquisitions and consumption strategies

Build and incorporate automated unit tests, participate in integration testing efforts

Work with teams to resolve operational & performance issues

Work with architecture/engineering leads and other teams to ensure quality solutions are implemented, and engineering best practices are defined and followed.

Tech Stack

Apache Spark

Apache Spark Streaming using Apache Kafka

Apache Hive

Apache Airflow

Python

AWS EMR and S3

Snowflake

SQL

Other Tools & Technologies :: PyCharm, Jenkin, Github.

Apache Nifi (Optional)

Scala (Optional)