Senior Data Engineer

Overview

Hudson Valley Credit Union is currently recruiting for the position of Senior Data Engineer. Primary Functions: Design, build, and optimize our data infrastructure, leveraging the power and scalability of Databricks within our Azure environment. Maintain a deep expertise in Databricks and its ecosystem in order to build high-performance data pipelines, ensure data quality, and enable advanced analytics and machine learning initiatives.Responsibilities

Design, develop, and maintain robust and scalable data pipelines primarily using Databricks, leveraging Spark (PySpark or Scala), Delta Lake, and Databricks Workflows

Build and optimize data lakehouse solutions on Azure Data Lake Storage Gen2 integrated with Databricks

Implement and enforce data quality frameworks and data governance policies within Databricks using tools and features like Delta Lake's ACID properties and data validation techniques

Identify and resolve performance bottlenecks in Databricks-based data pipelines and Spark applications

Collaborate closely with analysts to understand their data requirements for advanced analytics, providing them with curated and performant datasets within Databricks

Automate data ingestion, transformation, and deployment processes, potentially integrating with CI/CD tools and Databricks MLOps

Provide technical leadership and mentorship to junior data engineers on Databricks best practices and performance tuning

Document Databricks notebooks, data pipelines, data models, and configurations

Ensure data security and compliance within the Databricks environment, adhering to relevant security standards and best practices

Troubleshoot and resolve complex data-related issues within the Databricks ecosystem

Work independently and contribute to immediate and cross-functional teams

Participate in design discussions

Analyze problem statements, break down problems, and provide solutions

Follow an agile-based approach to deliver data products

Take ownership of large tasks, delivering on time, and mentoring team members

Oversee the maintenance and enforcement of SDLC and ITSM processes, change management procedures, departmental standards, style requirements and all other procedures necessary for data and system integrity

Adhere to all Credit Union policies, procedures, and regulatory agency requirements. Participate in all required and recommended training and development including, but not limited to, Bank Secrecy Act training (BSA) and demonstrate attained knowledge. Participate on Credit Union teams, projects and strategic initiatives when the opportunity arises. Perform additional duties and special projects as assigned.

Embrace and apply HVCU's guiding principles to all activities and responsibilities. This includes the Credit Union's Mission, Vision, Core Values, Employee and Member Value Propositions, Sales and Service Model, and commitment to Lean Six Sigma practices. Support the Credit Union's initiatives by demonstrating teamwork and professionalism.

Responsible for regular and predictable attendance including punctuality.

Qualifications

Bachelor's Degree or equivalent in Computer Science or related field required.

Master's Degree or equivalent in Computer Science or related field preferred.

License/Certification Databrick Data Engineer Associate required.

License/Certification Databrick Data Engineer Professional preferred.

License/Certification Mircrosoft Azure Data Engineer preferred.

Minimum 5 years of experience as Data Engineer required.

Minimum 5 years of experience Data Wharehousing, Data Lakes and big date platforms required.

Minimum 5 years Python experience required.

Minimum 3 years Databricks experience required.

Click here to view full job description

Why choose HVCU?

HVCU cares for our employees, members, and community. We actively seek to create an inclusive workforce by hiring people with diverse perspectives and experiences. We celebrate and embrace what makes each of us different and foster an environment where everyone feels that they belong. In commitment to our diversity, equity, and inclusion mission, we encourage members of historically underrepresented communities to apply, including those who identify as women, minorities, veterans, disabled and LGBTQ+.

Employee Perks:

HVCU University & Academy; Professional Development Training

Student Loan Repayment & Tuition Reimbursement Programs

Medical, Dental, and Vision coverage (coverage starting as low as $20 per month)

401(k) with employer match and non-elective employer contribution

Paid time off (PTO), holidays, and community volunteer time

Life, Short-Term, and Long-Term Disability Insurance

Discounted Loan Rates & Fees

Hudson Valley Credit Union is an AA/EEO employer committed to equal opportunity and employee diversity. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, sexual orientation, gender identity, the status of being transgender, national origin, disability, military or veteran status, genetic information, marital or familial status, domestic violence victim status, or any other characteristic protected by law.