Data Engineer

Responsibilities:

Design, develop, and maintain scalable data pipelines for the efficient collection, storage, and processing of large datasets.

Develops and design best practices around data collection and maintain orchestration of data flows, for batch, near-real time data.

Develop and design best practices of CDC and SCD for existing systems to maintain data consistency

Participates in evaluation and deployment of big data tools, relevant to the development.

Ability to identify different platforms, techniques, and methodology used in big data platforms, and demonstrate for various system in the organization.

Optimize and tune data systems for performance and reliability.

Develop and implement data models and schemas to support business intelligence and analytics needs.

Monitor and troubleshoot data pipelines to ensure smooth and efficient operation.

Stay up-to-date with the latest advancements in data engineering and incorporate them into our data infrastructure.

Document data engineering architectures and follow best practices.

Qualifications:

Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field.

5+ years of experience as a Data Engineer, Data Architect, Big Data Administrator or similar role.

Strong programming skills in Python.

Experience with UNIX system administration using command line.

Experience with data processing frameworks such as Airflow, Airbyte, DBT, Hadoop, Spark, Hive, and Kafka,

Proficiency in SQL and experience with relational databases.

Knowledge of data modeling, ETL processes, and data warehousing solutions.

Knowledge of Structuring data in various schemas suitable for a data lake, warehouse, or marts.

Experience in implementing scalable, reliable, and efficient data storage solutions with PostgreSQL, MinIO.

Experience in enhancing the performance of big data applications, including optimizing system operations.

Excellent problem-solving skills and the ability to work independently or as part of a team.

Strong communication skills and the ability to explain complex technical concepts to non-technical audiences.