Data Engineer

Job Description

We are looking for a talented Data Engineer to join our team in Somerset, New Jersey. This is a Contract-to-Permanent opportunity where you will play a key role in designing, developing, and optimizing scalable data solutions. The ideal candidate will collaborate with cross-functional teams to modernize our data infrastructure and support the migration to an Azure Lakehouse architecture.

Responsibilities:

• Lead the migration of existing data assets to an Azure Lakehouse architecture, ensuring smooth transitions and optimized performance.

• Design, develop, and maintain scalable data pipelines using Azure Databricks, incorporating Delta Lake and Parquet file formats.

• Collaborate with data architects to implement best practices in data modeling, partitioning, and storage optimization.

• Enhance query performance and streamline cost management within Azure Data Lake Storage and Databricks environments.

• Develop and implement data quality frameworks to ensure accuracy, reliability, and consistency of data.

• Integrate analytics and reporting solutions on top of the data lake to support business intelligence needs.

• Contribute to the roadmap for transitioning to Microsoft Fabric, leveraging Delta Lake and Parquet assets.

• Document workflows, architecture designs, and operational processes to ensure seamless knowledge sharing.

• Mentor team members with less experience and lead technical training sessions to foster skill development.• Bachelor’s degree in Computer Science, Information Systems, or a related field, or equivalent relevant experience.

• At least 5 years of experience in data engineering with a strong focus on Azure data platform technologies.

• Advanced knowledge of Azure Databricks, Delta Lake, and Parquet file formats.

• Hands-on experience in building and maintaining data pipelines and implementing data modeling strategies.

• Proficiency in programming languages such as Python or Scala.

• Solid understanding of data governance, security, and compliance standards.

• Familiarity with Microsoft Fabric architecture and migration strategies.

• Excellent communication and collaboration skills in team-oriented environments.