Azure Data Engineering Architect

Description

Lead Azure Data Engineer Architect role is to strategically design, develop and implement Azure Cloud based Data Warehouse, Data Lake, Data Lakehouse, Data Marts & BI solution using modern data engineering techniques and advanced analytics methods to provide information & insights to end users. The role requires modeling, collecting, harmonizing, aggregating, storing and reconciling data from variety of data sources for our petroleum refineries IT/OT technology platform including time series Database, SCADA data, batch, real time, enterprise applications, Rest API’s, streaming & IOT data. The role will help design and build data pipelines, streams, data warehouse, data lake, reporting application, data generators and a whole range of tools to provide information and insight.

The ideal candidate will be responsible for designing, developing, and maintaining scalable data processing pipelines in a cloud-based environment. The candidate should have a deep understanding of Azure ADF, Databricks, streaming analytics, data engineering best practices, and a proven track record of delivering high-quality real time solutions.

Responsibilities

Strategy & Planning

Assess and cultivate long-term strategic goals for data warehouse, data lake, BI & advance analytics development in conjunction with end users, managers, clients, and other stakeholders.

Coordinate and work with other technical staff to develop Azure Data & BI architecture, coding standards, and quality assurance policies and procedures.

Promote a culture of self-serve data analytics by minimizing technical barriers to data access and understanding.

Is responsible for data collaboration and communication with various stakeholders.

Operational Management

Lead the integration efforts for Azure data warehouse, data lakehouse - sourcing data from disparate enterprise systems including applications both on premises & cloud, IOT devices, Data Service (Rest API’s), time-series database, semi-structure, unstructured data & external data sources.

Organize and lead projects in the implementation and use of BI software tools and systems.=

Design, code, test, and document all new or modified BI systems, applications, and programs.

Develop the semantic layer, metadata, reports, and report definitions.

Assist in the design of databases to ensure interoperability with DW & BI solutions.

Produce ELT/ETL design guidelines to ensure a manageable ELT/ETL infrastructure for the analytics system.

Participate in design discussions about new features and ensure real-time technology integrates seamlessly with other pieces of the platform

Applying technical knowledge to architect solutions that meet business needs, AI/ML roadmaps, and contributing to architectures that enable the ability to scale to support additional modelling use cases (e.g. Azure ML, Azure Databricks, Azure EventHub/IoT Hub, etc)

Build processes supporting data transformation, data structures, metadata, dependency and workload management.

Work with project managers to ensure that data entry, retrieval, change, and delete functions meet business requirements for project completion.

Act as the primary owner and maintainer of new and existing Lakehouse/DW/BI data models

Design, implement, and maintain CI/CD pipelines using Azure DevOps to integrate analytical solutions into cloud based custom developed applications

Support Cloud strategy team to integrate analytical capabilities into an overall cloud architecture and business case development

Facilitate Master/ Reference Data Management and overall Data Governance process

Conduct job duties and responsibilities according to the organization’s business systems development methodology and its Systems Development Life Cycle (SDLC) methodology.

Design and deliver end-user training and training materials; provide technical support as necessary.

Troubleshoot Data Integration & BI tools, systems, and software; performance-tune these applications as necessary.

Defines and lead proof of concept activity associated with cloud data & analytics technology assessment.

Act as evangelist for Azure Cloud data & analytics benefits across the organization; increase usage to relevant departments.

Act as a subject matter expert on all things cloud data architecture and pipeline development

Position Requirements

Knowledge & Experience

Minimum of 12 to 15 years of Data Engineering, Data Warehouse/ Business intelligence with hands-on experience leading and delivering multiple full life cycle implementations.

Comprehensive data warehouse analysis and design experience, with full knowledge of data warehouse methodologies and data modeling.

Hands-on experience & deep understanding in designing, development and implementation of Azure Data & Analytics architecture, data modeling, transformations, and development.

In-depth knowledge and minimum 5 to 7 years’ experience developing & implementing data engineering solutions using Azure services for Data Warehouse, Data Lakehouse, & Advance analytics- Primarily- Azure Data Factory, ADLS, Azure Databricks, Azure Functions, Logic Apps, IoT Hub, Event Hub & Azure SQL

Experience with Data pipeline and workflow management tools & Azure DevOps processes.

Strong proficiency in programming languages such as Python, PySpark, or Scala

End to End Implementations expertise in Azure Data Warehouse & Data Lakehouse

Solid understanding of database structure principles, Master and Reference Data management

Experience & expertise in optimizing the ETL/ELT data pipeline technology options to minimize the on-going operational cost of Azure data engineering & storage solutions.

8+ years experience in agile Data warehouse development methodology.

8+ years’ experience designing and developing dimensional models for data warehousing

3+ years’ experience in MSBI suite primarily SSIS

3+ years of implementing operational data model is plus not essential

3+ years of experience with big data batch/streaming processing.

5+ years’ experience using Pyspark, Python or Scala

Experience delivering reporting solution using visualization tools like PowerBI

Experience in implementing SAP data analytics solution in Azure is plus not essential

Experience using data modeling tool like ERWin

Ability to lead and work with remote & offshore teams.

Experience integrating & analyzing data from time-series databases, SCADA data, IOT, voice to text, images & videos for creating Industrial Intelligence solution.

Knowledge/expertise on Azure ML/AI predictive modeling capabilities is a plus

Experience using proven methods to solve business problems using Azure Data and Analytics services in combination with building data pipelines, data streams and system integration

Hands-on experience with analytics and big data technologies within Microsoft Azure Cloud platform in tools such as Azure Machine Learning & Azure Cognitive Services is plus not essential

Experience with data profiling, cataloguing, and data mapping for technical design using a use case-based approach that drives the construction of technical data flows

Experience with Azure Administration & Security (Connection setup, User setup, Object security) is plus

Understanding of statistical and machine learning modeling with experience applying these modeling techniques to business problems is a plus not essential.

Ability to set up data and experimental platforms using Azure cloud service

Ability to work independently with business stakeholders to collect and understand requirements for reporting & analytics.

Full project management and software development life cycle experience.

Experience with data process flowcharting techniques.

Oil & Gas industry experience preferred but not essential.

Experience & knowledge of SAP S4 and SAP BW4HANA is a plus not essential.

Excellent quantitative, analytical & mathematical skills is a plus

Strong understanding of the organization’s goals and objectives

Formal Education & Certification

Bachelor’s degree in computer science, information systems, or a related field, or equivalent work experience.

Certifications in Azure Data Engineering and Analytics space is a plus

Personal Attributes

Ability to work independently and collaboratively in a fast-paced environment

Strong written and oral communication skills.

Strong presentation and interpersonal skills.

Exceptional analytical, conceptual, and problem-solving abilities.

Able to prioritize and execute tasks in a high-pressure environment.

Experience working in a team-oriented, collaborative environment.