Senior Data Engineer Python

An Azure Data Engineer needs core skills in:

- Creating effective and scalable data structures in SQL & Python

- Azure Cloud Fundamentals and ETL/Data Pipelines focusing on integrating, transforming, building and managing end-to-end data pipelines, combining data from several sources and storing data using services like ADLS Gen2

- Synapse Analytics

- Databricks

- Proficiency in administering Azure databases, and securing data, plus strong data modeling and big data tech (Spark)

Programming: Python (scripting, PySpark) & SQL (advanced querying, optimization).

Azure Fundamentals: Core cloud concepts, Resource Groups, Subscriptions.

Data Integration & ETL/ELT: Azure Data Factory (ADF).

Data Storage: Azure Data Lake Storage (ADLS Gen2), Azure SQL Database.

Big Data Processing: Azure Databricks (Spark, PySpark) for large datasets.

Data Warehousing: Azure Synapse Analytics (serverless pools, data warehousing).

Foundational Knowledge

Data Modeling: Designing efficient data structures (star/snowflake schemas).

Data Architecture: Understanding data lakes vs. warehouses.

Security: Data encryption, access control, compliance within Azure.