Data Engineer

Who are we?

About US

Banyan Cloud is a US-based Cyber Security Product Development Company, headquartered in San Jose, California, USA, which owns the SaaS product “Banyan Cloud”, First of its kind Cyber Security Platform, which simplifies the code to cloud security for multi cloud and multi technologies.

It's a once-in-a-lifetime opportunity to join our rocket ship start-up run by a world-class executive team. We are looking for candidates that aspire to be a part of the cutting-edge solutions and services we offer that address next gen data evolution challenges.

This role will be responsible to build a highly scalable, available, and secure data infrastructure. The Data Engineer will work with software engineers and data science engineers to build and support various data initiatives of multiple services to ensure optimal data delivery architecture is consistent throughout ongoing projects.

Job Responsibilities:

Create and maintain optimal data pipeline architecture for data intensive applications.

Ability to understand the business requirements, system architecture and process guidelines.

Ability to code for multiple projects at a time.

Ability to code for complex requirements under tight timelines.

Ability to work with cross-functional / domain teams.

Ability to handle end-to-end projects on an individual basis in fast-paced environment.

Build, maintain and document data pipelines from scratch.

Design and build scalable, reliable, and cost-effective data pipelines on the cloud using the latest technologies such as AWS, Azure, or Google Cloud Platform.

Develop and maintain ETL workflows and data processing pipelines using tools such as Apache Spark and Kafka.

Develop and maintain ETL pipelines using cloud ETL technologies like AWS Glue, Google Dataflow, Azure Data Factory

Collaborate with data scientists and analysts to ensure that data is accurate, reliable, and accessible.

Implement and maintain data storage solutions on the cloud, such as data lakes, data warehouses, or NoSQL databases like MongoDB, DynamoDB.

Optimize data infrastructure performance and cost by monitoring usage, identifying bottlenecks, and recommending improvements.

Job Requirements:

3 to 5 years of experience with large-scale data processing and data pipelines.

Programming experience on Python, Shell scripting, PySpark, and other data programming languages.

Interaction with SQL as well as No SQL Databases (Mongodb), data warehouses, data lakes

Experience with ETL/ELT tools such as Hadoop or Spark SQL tools such as Spark SQLQuery

Experience with AWS/GCP/Azure web services like AWS Glue, S3, RDS, Azure Data Factory, Blob Storage, Google Data Flow)

Creation of Web Services / APIs in Python

Experience working with time series data

Experience in data privacy and security-related projects

Added advantage:

Familiarity with regression and clustering algorithms

Experience with classification algorithms like decision trees, random forests and XGBoost

Extensive Experience working with Pandas, sci-kit learn, NumPy

NLP, Reinforcement learning, Outlier detection, Neural networks

Experience in Files Processing - XML, CSV, Excel, Json, etc Formats

Knowledge of Structured and Non-Structured Data processing

Knowledge of Distributed Big Data Processing

Experience with ‘TensorFlow’ is a plus.

Banyan Cloud provides equal employment opportunity (EEO)

Banyan Cloud provides equal employment opportunity (EEO) to all persons regardless of age, color, national origin, citizenship status, physical or mental disability, race, religion, creed, gender, sex, pregnancy, sexual orientation, gender identity and/or expression, genetic information, marital status, status with regard to public assistance, veteran status, or any other characteristic protected by federal, state or local law. If you are unable to submit an application because of incompatible assistive technology or a disability, please contact us at