Data Engineer with Databricks & Spark Experience
We are looking for a data engineer who has strong experience in building scalable and reliable data pipelines using Databricks and Spark. You will be working with various data sources and formats, and transforming them into valuable insights for our business.
Responsibilities
Design, develop, and maintain data pipelines using Databricks and Spark, and other cloud technologies as needed
Optimize data pipelines for performance, scalability, and reliability
Ensure data quality and integrity throughout the data lifecycle
Collaborate with data scientists, analysts, and other stakeholders to understand and meet their data needs
Troubleshoot and resolve data-related issues, and provide root cause analysis and recommendations
Document data pipeline specifications, requirements, and enhancements, and communicate them effectively to the team and management
Create new data validation methods and data analysis tools, and share best practices and learnings with the data engineering community
Implement ETL processes and data warehouse solutions, and ensure compliance with data governance and security policies
Qualifications
Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent work experience
5+ years of experience in data engineering with Databricks and Spark
Proficient in SQL and Python, and familiar with Java or Scala
Experience with cloud platforms, such as Azure or AWS
Experience with big data technologies, such as Kafka, Hadoop, Hive, etc.
Experience with data warehouse and data lake concepts and architectures
Experience with data integration and ETL tools, such as Azure Data Factory or Talend
Experience with data visualization and reporting tools, such as Power BI or Tableau
Strong analytical and problem-solving skills
Excellent communication and teamwork skills