Valiance Solutions - Big Data Engineer - Python/Hadoop/Hive

About Us : Valiance is a global AI & Data analytics firm helping clients build cutting-edge technology solutions for digital transformation.

We work with some of the marquee brands across India, US and APAC to build transformative solutions for Credit Risk, Fraud, Predictive Maintenance, Quality Inspection, Data lake, IOT analytics etc.

Our team comprises 120 professionals across Machine Learning, Data Engineering & Cloud expertise We are seeking a skilled and experienced Big Data Engineer to join our team.

The ideal candidate should have a minimum of 3 years of experience in big data engineering and possess strong expertise in Python, SQL, Pyspark, Hadoop, Hive, Delta Lake, and Airflow.

Responsibilities : - Design, develop, and maintain large-scale data processing systems - Implement data pipelines and ETL processes - Collaborate with data scientists and analysts to understand data requirements - Optimize and tune the performance of big data applications - Troubleshoot and resolve issues in data pipelines and processing systems - Ensure data quality and integrity throughout the data lifecycle - Stay updated with the latest big data technologies and best practices Requirements : - Bachelor's or Master's degree in Computer Science, Engineering or a related field - Minimum of 3 years of experience in big data engineering - Proficiency in Python, SQL, Pyspark, Hadoop, Hive, Delta Lake, and Airflow - Strong understanding of distributed computing and big data technologies - Experience with data warehousing, data modeling, and schema design - Familiarity with cloud platforms such as AWS, Azure, or GCP is a plus - Excellent problem-solving and analytical skills - Strong communication and collaboration abilities - Ability to work effectively in a fast-paced and dynamic environment (ref:hirist.tech)