Python, Data Engineer, AWS

Python, Data Engineer, AWS

Key responsibilities

- Develop and maintain data pipelines Build, test, and maintain ETL (Extract, Transform, Load) processes to move and transform data.

- Design and manage data architecture Construct and manage data infrastructure, including data warehouses, data lakes, and databases.

- Automate data processes Use Python to automate tasks and build data processing applications.

- Integrate data sources Connect internal and external data sources and APIs, and ensure system compatibility.

- Ensure data quality and security Implement data quality checks, security controls, and access management policies.

- Collaborate with stakeholders Work with data scientists, analysts, and other teams to understand requirements and deliver solutions.

- Optimize systems Monitor system performance, troubleshoot issues, and implement optimizations for reliability and efficiency.

Required skills and qualifications

- Programming Proficiency in Python is a must. SQL and other languages like Java or Scala are also valuable.

- Databases Strong knowledge of database management systems (relational and NoSQL) and data modeling.

- Big Data Technologies Experience with big data frameworks such as Apache Spark and Hadoop.

- Cloud Platforms Familiarity with cloud services is often required, such as AWS, GCP, or Azure.

- Version Control Experience with version control systems like Git.

- Problem-Solving Strong analytical and problem-solving skills.

- Communication Excellent communication skills to collaborate with technical and non-technical teams.

- Education A bachelors degree in Computer Science, Engineering, or a related field is typically required.