Senior Data Engineer

We are seeking a highly skilled and experienced Senior Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will collaborate closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will possess deep expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop, Spark, and more.

Key Responsibilities:

Design, Develop, and Maintain Data Pipelines: Leverage Big Data technologies (e.g., Hadoop, Spark, Python, Scala) to design and maintain large-scale data processing systems.

Implement Scalable Data Solutions: Work on data ingestion, transformation, storage, and analysis solutions that ensure scalability, efficiency, and reliability.

Stay Current with Industry Trends: Keep updated on industry trends and emerging technologies to continuously improve the data architecture.

Collaborate with Cross-functional Teams: Partner with teams to understand business requirements and translate them into effective technical solutions.

Optimize Data Pipelines: Enhance the performance, scalability, and reliability of existing data pipelines.

Ensure Data Quality: Develop and implement automated testing frameworks for continuous quality assurance in data pipelines.

Unit & Integration Testing: Conduct extensive testing to ensure robustness and accuracy across all data workflows.

Monitor Production Environments: Troubleshoot and resolve issues in live data pipelines, ensuring minimal disruptions.

Required Qualifications:

Education: Bachelor's degree in computer science, Information Systems, or related discipline (master’s preferred).

Experience: Minimum 5 years of relevant experience in Big Data Engineering or equivalent training.

Technical Expertise:

Big Data Technologies: Strong experience with Hadoop, Spark, Hive, and Trino.

Cloud Technologies: Proficiency with AWS services like S3, EMR, Glue, Lambda, Athena, etc.

Programming: Solid understanding of Python, Java, or Scala for developing high-performance, scalable code.

SQL Skills: Advanced proficiency in SQL (window functions, joins, aggregations).

Spark Performance Tuning: Knowledge of Spark's architecture and performance optimization techniques.

Experience in developing enterprise-grade solutions in Agile or iterative environments.

Communication Skills: Excellent verbal and written technical communication, with the ability to collaborate across teams.

Essential Technical Skills:

Big Data Technologies:

Hands-on experience with Hadoop, Spark, Hive, Trino.

Understanding of data challenges such as data skew, handling massive data volumes, and troubleshooting job failures due to resource limitations.

SQL Expertise:

Extensive use of window functions, multi-table joins, and complex aggregations. Ability to write/optimize SQL queries on the fly.

Apache Spark:

Familiarity with Spark architecture (executors, tasks, stages, DAG) and performance tuning (e.g., partitioning, caching, broadcast joins).

Hands-on experience with troubleshooting and optimizing Spark jobs.

Cloud Technologies:

Exposure to AWS services like S3, EMR, Glue, Lambda, and others.

Experience with EKS and Serverless architectures is a plus.

Programming Skills:

Proficiency in Python, Scala, or Java for modular and performant code development.

Strong foundation in functional programming concepts such as immutability, higher-order functions.

Additional Skills:

Production Data Pipelines/ETL Systems:

Experience managing production-grade data pipelines and ETL systems.

CI/CD Knowledge:

Familiarity with continuous integration and deployment processes.

Test Case Writing:

Hands-on experience with writing and maintaining unit, integration, and end-to-end tests.

AWS Certifications (Good to Have):

AWS certifications will be a significant advantage.

Preferred Qualifications:

Experience in Financial Services industry.

Demonstrated ability to work in a fast-paced environment.

Ability to handle multiple competing priorities effectively.

This enhanced JD integrates the hiring manager's comments, with a stronger focus on the key technical skills (Big Data, AWS, Spark, etc.), programming languages, and the critical work environment (hybrid, East Coast candidates, etc.). It also emphasizes the technical certifications and education requirements, ensuring alignment with the job's needs.