Big Data Engineer

Job Description

Company Overview: Welcome to the forefront of big data innovation! At our company, we're leading the charge in harnessing the power of big data to drive insights and innovation. Our mission is to develop scalable and robust big data solutions that empower organizations to extract value from their data at scale. Join us and be part of a dynamic team committed to shaping the future of big data engineering.

Position Overview: As a Big Data Engineer, you'll play a pivotal role in designing, building, and maintaining our big data infrastructure and platforms. Working closely with cross-functional teams of data scientists, analysts, and software engineers, you'll ensure the reliability, scalability, and efficiency of our big data pipelines and systems. If you're passionate about big data engineering and eager to drive innovation through scalable data solutions, we want you on our team.

Requirements

Key Responsibilities:

Big Data Infrastructure Design: Design and architect scalable and reliable big data infrastructure and platforms, including data lakes, data warehouses, and streaming data pipelines.

Data Ingestion and Processing: Develop and maintain data ingestion pipelines to collect and process large volumes of data from diverse sources, ensuring data quality and integrity.

Data Storage and Management: Implement and manage big data storage solutions, including distributed file systems (e.g., HDFS), NoSQL databases (e.g., Cassandra, MongoDB), and cloud storage services (e.g., Amazon S3, Google Cloud Storage).

Data Transformation and ETL: Design and implement data transformation and ETL (Extract, Transform, Load) processes to standardize, cleanse, and enrich data for analysis and modeling.

Data Querying and Analysis: Develop and optimize data querying and analysis tools and frameworks, enabling data scientists and analysts to extract insights from large datasets.

Data Governance and Security: Implement data governance policies and security controls to ensure compliance with regulatory requirements and protect sensitive data.

Monitoring and Optimization: Monitor big data pipelines and systems for performance, reliability, and efficiency, proactively identifying and addressing issues to minimize downtime and optimize resource utilization.

Documentation and Collaboration: Document big data infrastructure and processes, including architecture diagrams, data lineage, and workflow documentation, and collaborate with cross-functional teams to ensure alignment and transparency.

Qualifications:

Bachelor's degree or higher in Computer Science, Engineering, or related field.

Strong background in big data engineering, with hands-on experience in designing, building, and maintaining big data infrastructure and platforms.

Proficiency in programming languages such as Java, Python, or Scala, and experience with big data processing frameworks such as Apache Hadoop, Apache Spark, or Apache Flink.

Experience with distributed computing technologies and concepts, including MapReduce, Spark RDDs, and streaming data processing.

Familiarity with big data storage solutions and databases, including HDFS, Cassandra, MongoDB, Amazon S3, and Google Cloud Storage.

Strong problem-solving abilities and analytical thinking, with a keen attention to detail and a passion for tackling complex technical challenges.

Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and communicate technical concepts to non-technical stakeholders.

Benefits

Competitive salary: The industry standard salary for Big Data Engineers typically ranges from $150,000 to $230,000 per year, depending on experience and qualifications. Exceptional candidates may be eligible for higher compensation packages.

Comprehensive health, dental, and vision insurance plans.

Flexible work hours and remote work options.

Generous vacation and paid time off.

Professional development opportunities, including access to training programs, conferences, and workshops.

State-of-the-art technology environment with access to cutting-edge tools and resources.

Vibrant and inclusive company culture with team-building activities and social events.

Opportunities for career growth and advancement within the company.

Exciting projects with real-world impact across diverse industries.

Chance to work alongside top talent and industry experts in the field of big data engineering.