Lead - Azure Databricks, Pyspark, SQL

Position: Lead - Azure Databricks

Overall IT experience: 9 to 13 years

Primary Skills: Azure Databricks, Hadoop, Hive, Pig, Impala, ETL, Data warehouse

Secondary Skills: U-SQL, Spark, Scala, Python

Work Location: Bengaluru, Pune, Mumbai, Noida, Gurugram

Mode of work: Hybrid at Infogain office locations mentioned above

Notice Period: 0 to 30 Days

Essential experience:

Experience in building ETL / data warehouse transformation processes.

Direct experience of building data pipelines using Azure Data Factory and Apache Spark (preferably Databricks).

Experience with other Open-Source big data products Hadoop (incl. Hive, Pig, Impala).

Evaluating the performance and applicability of multiple tools against customer requirements. Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.

Coding complex U-SQL, Spark (Scala or Python), T-SQL.

Developing Modern Data Warehouse solutions using Azure Stack (Azure Data Lake, Azure Data Factory, Azure Databricks).