Senior Data Engineer

Job Description

Senior Data Engineer

MUST be local to Iselin NJ, Hybrid Schedule at least 3 days onsite

Salary: up to $160k

No Sponsorship Available

Major Responsibilities:

Building and maintaining data pipelines to ingest data from source systems into Databricks

Designing, Building and maintaining generic frameworks to ingest data through the medallion layers

Working with the core data platform team to integrate the framework with reference architecture

Assisting the data mesh teams with best practices and onboarding to the new framework

Requirements:

10+ years of data engineering experience in building OLTP and OLAP databases, both on-prem and in the cloud

5+ years of experience with large scale data processing using SQL, Python, Pandas and PySpark

5+ years of experience building robust and scalable data pipelines in Databricks Lakehouse platform on any key cloud platform provider, preferably Azure cloud

Experience with implementing Medallion architecture for data processing

Experience with implementing Streaming data ingestion using Kafka

Test driven development experience

Proven experience in solution design and applying best design practices

Data modelling and ER/UML diagram skills

Experience with orchestration/enterprise scheduler platforms like Airflow, Tidal etc.

Experience with implementing API based data integration pipelines

Experience with source control and DevOps for CI/CD

Experience working in Agile/scrum methodologies

Experience with Jira, confluence for project work and documentation

Experience with Service now for incident management

Finance domain knowledge in Investment Banking (Securities, Capital markets)and Security reference data knowledge

Nice to have ADF and SSIS experience* College degree required.

Full-time

Hybrid remote