Senior Data Engineer

Experienced Senior Data Engineer (12+ Years) to support large scale data platform modernization initiatives within a regulated banking environment.

The role focuses on designing and building enterprise-grade in-house frameworks, supporting high-volume batch and CDC-based incremental processing using the Cloudera platform, and enabling ongoing Google Cloud Platform (Google Cloud Platform) modernization efforts.

Technology & Skill Requirements

Apache Spark (PySpark and/or Scala) in large-scale production environments

Cloudera Hadoop ecosystem (HDFS, Hive, YARN, Spark on Cloudera)

Strong SQL expertise with complex transformations, performance tuning, and reconciliation logic

Enterprise RDBMS experience with Oracle and MS SQL Server

Batch ingestion, incremental ingestion, and CDC processing patterns

CDC concepts and tooling (tool-agnostic: GoldenGate, Debezium, or equivalent)

Data merge, deduplication, watermarking, checkpointing, and SCD handling

Google Cloud Platform services including Dataproc, Composer and Dataplex

Hybrid on prem to cloud data architecture and migration patterns

Metadata-driven framework development and data quality validation techniques

CI/CD pipeline implementation using enterprise tooling (GitHub Actions, Jenkins, DevOps)

Git-based development workflows, code reviews, and automated testing practices

Experience using Copilot or similar AI-assisted development tools safely and effectively in enterprise environments

Logging, monitoring, alerting, and operational readiness practices

Secure coding, access control, and compliance-aware development

Documentation of design artifacts, runbooks, and operational procedures