Azure Data Engineer

Job Description

We are looking for a Senior/Principal Azure Data Engineer to design, build, and maintain scalable, reliable, and cost-effective data pipelines and data products using Microsoft Azure technologies.

Key Responsibilities

Design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using Azure Data Factory, Azure Synapse Analytics, Azure Databricks, and Azure Data Lake Storage Gen2.

Work with storage solutions such as Azure Blob Storage for data ingestion, processing, and storage.

Build and optimize big data processing pipelines using Apache Spark (PySpark preferred).

Ensure data quality, validation, governance, and reliability across data pipelines.

Collaborate with data scientists, analysts, architects, and stakeholders to deliver data solutions.

Optimize performance, scalability, and cost efficiency of data workloads in Azure.

Implement CI/CD pipelines, version control, testing, and deployment automation for data solutions.

Monitor, troubleshoot, and resolve data pipeline issues proactively.

Maintain technical documentation and best practices for data engineering standards.

Stay updated with emerging data engineering and cloud technologies.

Requirements

Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).

Strong experience in Azure Data Engineering and ETL development.

Proficiency in SQL using systems like SQL Server and PostgreSQL.

Strong understanding of software engineering principles (CI/CD, version control, testing).

Strong programming skills in Python (preferred) or Scala/Java/C#.

Experience working with file formats like Apache Parquet, Delta Lake, and Avro.

Strong problem-solving, analytical, and communication skills.

Preferred Skills

Technical leadership and stakeholder management

Experience working in consulting or client-facing environments

Strong learning agility and adaptability in cloud ecosystems

Experience building production-grade Spark applications using PySpark