Python Data Engineer Bengaluru, Karnataka, India

Python Data Engineer Bengaluru, Karnataka, India

Description

About the Role:

We are seeking a skilled Data Engineer to build and maintain our data infrastructure, enabling business intelligence, reporting, and analytics. You will be responsible for developing scalable data pipelines, integrating data from diverse sources, and ensuring data quality and accessibility. Expertise in Amazon Redshift, Python, Apache Airflow, dbt, API integration, and AWS is essential. You will collaborate with cross-functional teams to optimize our data architecture and deliver clean, high-quality data to the business.

Key Responsibilities:

Data Pipeline Development: Design, build, and maintain robust ETL/ELT pipelines using Python, Apache Airflow, and dbt to ingest, transform, and load data into Amazon Redshift.

Amazon Redshift Management: Design, optimize, and maintain Amazon Redshift clusters for efficient large-scale data handling.

API Integration: Develop solutions to integrate external APIs for data ingestion, ensuring proper data extraction, transformation, and integration.

Data Modeling: Design and maintain scalable data models in Redshift (star/snowflake schemas) to support analytics and reporting.

AWS Infrastructure Management: Utilize AWS services (S3, Lambda, EC2, CloudWatch) to build and maintain a scalable and cost-efficient data architecture.

dbt (Data Build Tool): Leverage dbt to manage and automate SQL transformations, ensuring modularity, reusability, and documentation.

Workflow Orchestration: Use Apache Airflow to orchestrate and automate data workflows and scheduled jobs.

Data Quality & Testing: Implement data validation checks and testing frameworks to ensure data integrity, accuracy, and compliance.

Collaboration: Work closely with data scientists, analysts, and product teams to understand data needs and provide technical solutions.

Performance Optimization: Tune SQL queries and manage Redshift cluster performance for efficient data access and analysis.

Data Governance: Enforce data governance policies to ensure compliance with security, privacy, and data quality standards.

Key Skills & Qualifications:

Bachelor's/Master's degree in Computer Science, Engineering, Data Science, or a related field.

3+ years of experience in data engineering with expertise in Amazon Redshift, Python, and AWS.

Strong experience with Apache Airflow.

Hands-on experience with dbt.

Proficiency in API development and integration (RESTful APIs).

Extensive experience with AWS services (S3, Lambda, EC2, RDS, CloudWatch).

Expertise in data modeling (star/snowflake schemas).

Advanced SQL skills for querying and optimizing large datasets in Redshift.

Experience building ETL/ELT pipelines and integrating data from multiple sources (structured/unstructured).

Familiarity with Git and best practices for code management and deployment automation.

Knowledge of data governance principles.

Preferred Qualifications:

Experience with real-time data processing tools (Kafka, Kinesis).

Familiarity with data visualization tools (Tableau, Looker, Power BI).

Knowledge of other data warehousing solutions (Snowflake, BigQuery).

Experience with DevOps practices (Docker, Kubernetes).

Understanding of machine learning pipelines.

Soft Skills:

Strong analytical and problem-solving skills.

Ability to work independently and collaboratively.

Excellent written and verbal communication skills.

Detail-oriented, proactive, and self-motivated.

Strong organizational and project management skills.

Requirements

Where are you located?

What's your nationality?

What is your expected salary for this job?

What is your notice period; when can you engage this job?

Please only apply and press yes if you have read the requirements, education, experience and certifications, and match these.

Location

Bengaluru, Karnataka, India