Data Engineer
Description:
3-4+ years of Data Engineering experience.
Strong Azure and Databricks
Strong SQL Skills.
Resume Keywords: Azure, Spark, Delta Lake, CI/CD, ABS, Databricks, PySpark
Required Skills & Experience
• Experience with Databricks platform on Microsoft Azure
• Advanced SQL Skills
• 4-6 years of hands-on data engineering experience
• 3-5 years of hands-on experience with processing large data sets required
• 3-5 years of hands-on experience with SQL, data modeling, relational databases and/or no SQL databases required
• Bachelor’s Degree in related field or equivalent work experience required
• Experience with processing large datasets using Hadoop, Spark, Kafka or similar distributed systems
• Experience with PySpark and distributed data processing frameworks
• Understanding of cloud computing technologies, business drivers and emerging computing trends
• Experience with ingesting various source data formats such as JASON, Parquet, Sequencefile, Cloud databases, MQ, Relational databases such as Oracle
• Working knowledge of Object Storage technologies to include but not limited to Data Lake Storage Gen2, ADLS etc.
• Working knowledge of unity catalog and data governance principles
• Working knowledge of Agile development /SAFe, Scrum and Application Lifecycle Management
• Strong background with source control management systems (GIT or Subversion); Build Systems (Maven, Gradle, Webpack); Code Quality (Sonar); Artifact Repository Managers (Artifactory), Continuous Integration/ Continuous Deployment (Azure DevOps)
• Experience with Python and SQL
• Creating and maintaining ETL processes and data pipelines
• Knowledgeable of best practices in information technology governance and privacy compliance
• Knowledge on auto finance industries core business functions
• Ability to establish priorities and meet tight deadlines.
• Ability to interact collaboratively with internal customers
• Demonstrated quantitative skills and data analytical skills
• Solid design, coding, testing and debugging skills