Data Engineer - DataBricks

Job Title: Data Engineer

Location: Anywhere in Canada (100% remote)

Duration: 12+ months

Position Overview:

We are seeking a highly skilled and motivated Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, with expertise in DataBricks and Data Lake technologies. This role will involve designing, building, and optimizing data pipelines and analytical solutions to support our organization's data-driven initiatives.

Key Responsibilities:

Design, develop, and maintain scalable data pipelines using DataBricks for data ingestion, transformation, and processing.

Implement and optimize DataBricks jobs to handle large volumes of data efficiently.

Collaborate with data architects and analysts to design and implement data lake solutions that meet business requirements.

Develop and maintain data lake infrastructure on platforms such as AWS S3, Azure Data Lake Storage, or Google Cloud Storage.

Write complex SQL and Spark queries for data analysis, aggregation, and reporting within the DataBricks environment.

Implement data governance, security, and access control policies to ensure data quality and compliance.

Monitor and troubleshoot data pipeline issues, identify performance bottlenecks, and implement optimizations.

Stay updated with the latest DataBricks features, best practices, and data engineering trends.

Qualifications:

Bachelor's degree in Computer Science, Engineering, or related field.

Proven experience working as a Data Engineer or similar role.

Strong proficiency in DataBricks and Apache Spark for big data processing.

Experience working with Data Lake architectures and technologies.

Solid understanding of cloud platforms such as AWS, Azure, or Google Cloud Platform.

Proficiency in SQL for data querying and manipulation.

Experience with data modeling, ETL processes, and data warehousing concepts.

Strong problem-solving and analytical skills.

Effective communication and collaboration abilities.

Preferred Qualifications:

Master's degree in Computer Science, Engineering, or related field.

Certifications in DataBricks or relevant cloud platforms.

Experience with other big data technologies such as Hadoop, Kafka, or Apache Flink.

Familiarity with containerization technologies such as Docker and Kubernetes.