Senior Data Engineer

Responsibilities

We are looking for a for a strong self-driven Senior Data Engineer to join the team of data and software engineering

Primary responsibilities would include:

Investigate source data systems to build comprehensive data models to serve as Single Source of Truth

Design and implement flexible, robust & scalable ETL infrastructure

Deploy pipelines in higher environments after robust testing criteria

Create and maintain robust Big Data pipelines for Machine Learning systems

Optimize performance of Big Data scripts

Create monitoring and reporting infrastructure for Machine Learning systems

Develop systems that facilitate Data Science activities such as feature extraction and model testing

Design and build automated systems with pipelines, transformations, data quality analytics and control logic.

Generate, process, log and correlate events across multiple sources.

Qualifications

Basic qualifications:

Bachelor’s degree in Computer Science, Engineering, or closely related field

2+ years of experience designing and developing software.

Proficient with Python, test-driven development, refactoring

Strong scripting skills to perform data/file manipulation

1+ years of experience with data aggregation platforms based on technologies such as Spark, SQL, Azure Data Lake, etc.

Solid understanding and proven skills in raw and processed stream design, relational database design and dimensional models

Familiarity with machine learning and predictive analytics methods.

Experience using scalable data ingestion and transformation systems in batch and near real time environments.

Experience with SQL, relational modeling, and big data tools such as Cosmos, Hive, Scope or Spark

Experience with ETL, data modeling, and working with Business Intelligence systems.

Solid understanding of event processing including publish/subscribe mechanisms.

Demonstrated ability to create and ship high quality code by using engineering best practices.

Strong logical reasoning, problem solving and debugging skills

Experience with developing software in distributed, cloud-based systems

Experience with Big Data systems, Data Pipelines

Technical experience with Python, SQL, other scripting languages.

Keen interest in Machine Learning

A team player with strong communication skills and drive for results

Preferred qualifications:

Azure, AWS

Python

SQL

Knowledge in Machine Learning