Spark Developer

Job Description

Spark 3 Developer

Preferred to Columbus, OH 3 days office is must for this role

Duration: 9 to10 months contract only

Job Summary

We are seeking a Specialist with expertise in Spark 3 Migration Specialist to join our team. The Specialist will be responsible for leveraging their knowledge and skills in quantum engineering to contribute to our projects and initiatives.

Responsibilities

This is migration project and client is moving to MapReduce to Spark 3

opportunity for a skilled consultant in conversion of Cascading/MapReduce to Spark

The Sr Software Engineer will be responsible for migrating Big Data workloads to Kubernetes/Spark 3.

Will lead the conversion of Cascading/MapReduce to Spark 3.

Will be touching the code daily within the IntelliJ IDE...doing Scala code reviews, making coding changes needed for the Scala/Spark upgrade, designing the new Kubernetes environment, etc.

Lead the end-to-end migration of legacy Big Data workloads from Cascading/MapReduce to Spark 3.

Re-engineer and refactor existing data processing pipelines into optimized Spark 3 jobs using Scala.

Implement and test Spark 3-based data flows ensuring functional and performance parity with the legacy system.

Experience

Spark 3, Scala, MapReduce, Cascading

Kubernetes, Docker, Helm

IntelliJ IDE

Hadoop Ecosystem, YARN, HDFS

CI/CD Tools (e.g., Jenkins, GitLab CI)

Monitoring Tools (e.g., Prometheus, Grafana)

Skills needed

This is migration project and client is moving to MapReduce to Spark 3

opportunity for a skilled consultant in conversion of Cascading/MapReduce to Spark

The Sr Software Engineer will be responsible for migrating Big Data workloads to Kubernetes/Spark 3.

Will lead the conversion of Cascading/MapReduce to Spark 3.

Will be touching the code daily within the IntelliJ IDE...doing Scala code reviews, making coding changes needed for the Scala/Spark upgrade, designing the new Kubernetes environment, etc.Company Description

Skill Demand Energy is a nationally certified minority-owned company with our Corporate Headquarters in Indianapolis, Indiana, and multiple offices across the U.S.

Skill Demand Energy partners with utilities, local and state governments, and other entities to provide solutions to reduce America's energy usage within the residential, commercial, and industrial and government sectors. We are a dynamic multiculturally owned firm fully committed to cocreate with our teams truly inclusive and equitable working environments that results in thriving diversity and professional success.

We are committed to hiring individuals that are representative of the communities our programs are designed to serve. Our work elevates co-creation through cultivating trust and acting with transparency. We recognize that representation matters.

Full-time