Data Engineer-Software Engineering

Job Description

Position Summary:

The IT Division is responsible for the development and operations of information systems for the State and Federal agencies doing business related to or using information from the administration of motor vehicles and driver licenses.

The Data Engineer position has core responsibilities for the design, development, maintenance, optimization, and operational support of components which move and ingest data from sources into the analytics systems.

Essential Duties and Responsibilities:

We are seeking a talented Data Engineer to join our team. You will work with cutting-edge technologies to build and maintain our data platform, focusing on vehicle and driver data solutions that improve road safety and reduce vehicle fraud. You will leverage the Microsoft data ecosystem, including SQL Server databases and data warehouses, Azure Synapse Analytics, and CosmosDB, to create efficient and reliable data solutions that power our analytics, reporting, and AI capabilities.

As a detail-oriented professional, you have a strong track record of independently managing projects and driving them to successful completion. Your exceptional business acumen and communication skills enable you to collaborate effectively with stakeholders, identify vital business questions, and make informed, data-driven decisions. You will play a key role in modernizing our SSIS and SSRS infrastructure while building complex data pipelines that process billions of events per year flowing through Azure Event Hubs. You will also have exciting opportunities to work on GenAI and LLM projects, building the data pipelines that power our AI initiatives.

Key responsibilities include:

Modernizing SSIS packages to cloud-native solutions using Azure Synapse Analytics

Migrating SSRS reports to modern Power BI dashboards and self-service analytics

Developing pipelines to and from SQL Server data warehouses

Developing complex SQL queries and stored procedures for vehicle and driver data analysis

Building sophisticated data pipelines to support road safety initiatives and vehicle fraud detection

Processing billions of events per year flowing through Azure Event Hubs

Designing and implementing scalable data pipelines using Azure Synapse Analytics and Apache Spark

Implementing high availability and performance solutions for mission-critical SQL Server databases and data warehouses

Establishing operational excellence with monitoring, alerting, and testing frameworks

Develop machine learning and anomaly detection algorithms

Direct Reports:

None

QUALIFICATIONS

Formal Education:

Bachelor's degree in Computer Science, Data Science, or related field. Equivalent work experience may be substituted.

Knowledge, Skills, and Abilities:

Basic Qualifications

3-5 years of hands-on experience in data engineering

Strong proficiency with TSQL, Azure Synapse Analytics and Apache Spark

Hands-on experience with Azure Event Hubs and stream processing

Experience building and maintaining data pipelines using Delta Lake

Proficiency in Python and SQL

Experience with version control systems (Git) and CI/CD practices

Preferred Qualifications

C# development experience, particularly with Azure Functions and .NET Core

Experience with modernizing SQL Server data warehouses to Delta Lake architecture

Experience migrating SSRS reports to Power BI dashboards and analytics

Knowledge of data modeling and dimensional modeling concepts

Familiarity with Azure Data Factory

Experience with real-time analytics and stream processing

Knowledge of data governance and security best practices

Experience with Power BI or similar visualization tools

Technical Stack

SQL Server

Core Technologies: Azure Synapse Analytics, Apache Spark, Delta Lake, Azure Event Hubs

Languages: SQL, PySpark, C#

Cloud Services: Azure Functions, Azure Data Factory, Azure Key Vault

Streaming: Azure Event Hubs, Spark Structured Streaming

Optional: Databricks, Lakehouse Architecture

Visualization: Power BI, Synapse Analytics Studio

Development: .NET Core, Azure DevOps

Disclaimer Statement: The preceding job description has been written to reflect management’s assignment of essential functions. It does not prescribe or restrict the tasks that may be assigned.

AAMVA is an Equal Opportunity Employer—M/F/D/V