Data Engineer

We are a company in Cloud and Infrastructure, Data Engineering, BI Analytics, AI and RPA. We are rapidly growing fast in the market and now we reached employee size of 800+ . We have offices in USA, Canada, 100+ Development Center in Thailand and 100+ Development Center in Chandigarh, Chennai,Hyderabad and Bangalore.

US HQ, 1000-strong, Technology and Business Process Company; $40Mn in revenues

2 Development Centre Chandigarh

Payroll :-MethodHub

Client: Infosys

Experience :4+

Design, develop, and maintain end-to-end data pipelines on AWS, utilizing serverless architecture.

Implement data ingestion, validation, transformation procedures using AWS services such as Lambda, Glue, Kinesis, SNS, SQS, and CloudFormation.

Write orchestration tasks within Apache Airflow.

Develop and execute data quality checks using Great Expectations to ensure data integrity and reliability.

Collaborate with other teams to understand mission objectives and translate them into data pipeline requirements.

Utilize PySpark for complex data processing tasks within AWS Glue jobs.

Qualifications:

Bachelor's degree in Computer Science, Engineering, or related field.

Strong proficiency in Python programming language.

Hands-on experience with AWS services.

Experience with serverless architecture and Infrastructure as Code (IaC) using AWS CDK.

Proficiency in Apache Airflow for orchestration of data pipelines.

Familiarity with data quality assurance techniques and tools, preferably Great Expectations.

Experience with SQL for data manipulation and querying.

Strong communication and collaboration skills, with the ability to work effectively in a team environment.

Experience with Data Lakehouse, dbt, Apache Hudi data format is a plus.