Data Engineer / Developer

Title: Data Engineer / Developer

Location: NYC, NY(Hybrid)

Duration: 12-24+ Months

Key Responsibilities

Primary duties are working in a team of data engineers and data architects to acquire data from internal and external sources, ingest into our repositories, and optimize it for data consumptions by end-users.

Implementing methods to improve data reliability and quality.

Combining raw information from different sources to create consistent formats.

Developing and testing architectures for data extraction and transformation.

Building and maintaining optimal data pipeline architectures.

Assembling large and complex data sets, as well as Big Data technologies.

Ensuring that data is readily available to data scientists, analysts, and other users.

Key Skills:

8+ years of experience in developing data acquisition, data ingestion, data transformation, file handling, encryption, sFTP protocol, ETL, ELT workloads on-prem and in the cloud.

8+ years of experience in writing SQL queries, database warehousing: normalization, query optimization, store procedures, views, etc.

8+ years of experience in Python, Java or .Net or C++ to build scripts, data movement applications and integration APIs.

Desired Qualifications:

Experience in using:

Snowflake.

APIs, containerization, and Kubernetes.

DevOps: Gitlab CI/ CD pipelines (both YAML and UI based), Gitlab runners, Terraform for CICD.

Snaplogic.

Perl/XML.

DBT, Data Build Tool.

REST APIs, GraphQL, RPC, gRPC, and Webhooks protocols.

Linux and shell scripting.

Designing and implementing event-driven architectures, messaging middleware, and microservices.

Azure cloud services for APIs, including Azure Functions, Azure API Management, and Azure App Service.

Containerization technologies, such as Docker and Kubernetes, and their deployments options on Azure: Azure Container App, Azure App Service, Azure Container Instances, and Azure Kubernetes Service.

Best practices for code quality, testing, and performance optimization.

NoSQL databases.

AI / ML concepts and implementations.

Gitlab Duo.