Amit Ranjan Big Data Engineer & Instructor

+971-**-***-**** ad40i3@r.postjobfree.com LinkedIn Profile Dubai, UAE PROFESSIONAL SUMMARY

A Senior Big Data Engineer and Tech Instructor with over a decade of industry experience. Proven track record of delivering impactful solutions to a dozen clients and organisations across e-commerce, finance, entertainment and transportation industries in diverse regions including the Middle East, Europe and Asia. Main areas of expertise cover:

• Data Management and Strategy

• Development of production-grade data platforms

• Design ETL processes

• Architect and implement data products

• Coding and analytical skills

• Optimisation and innovation

• Project Management

• Collaboration and mentoring

TECHNOLOGIES

Programming Languages: Python, Java, Scala

Big Data Technologies: Spark, Kafka, Airflow, Hadoop, MapReduce, Beam, Hive, ElasticSearch, Trino, Pig, Tez, Sqoop, NiFi, Debezium, HBase, Phoenix, MongoDB, Atlas, Luigi, Oozie, Quartz Scheduler Cloud Platforms: Google Cloud Platform (GCP), Amazon Web Services (AWS) Other:Docker, Jenkins, Shell Scripting, AtScale

EXPERIENCE

Senior Data Engineer Spotify Dubai, UAE 2020 – Till Date Tech Leader in the data team responsible for all the Ubiquity devices of Spotify including Car, Desktop, Smart Speakers.

● Led the data part of the initiative to generate better content recommendations, which resulted in ~1% uplift in the consumption. The initiative involved leveraging advanced analytics and machine learning to optimise content suggestions, significantly enhancing the overall user experience.

● Owned a major ML-backed service which serves over 1K req/s impacting 200M users daily. Implemented strategic improvements to elevate the service’s coverage and precision, resulting in a notable enhancement for over 60 million Monthly Active Users (MAU), equivalent to 12% of Spotify’s entire user base.

● Pioneered the development and management of strategic, high-quality data products, catering to diverse needs such as reporting, machine learning, and experimentation across the organisation. Ensured these products adhered to rigorous governance standards, contributing to a more agile and data-centric culture.

● Championed the growth and adoption of a data-centric culture by spearheading impactful data strategy initiatives and delivering targeted training sessions to multiple teams. Achieved a robust Net Promoter Score

(NPS), reflecting the positive reception and effectiveness of the data-driven strategies implemented.

● Optimised Apache Beam/Scio data pipelines, GCS and BQ storage footprints leading to heavy cost savings on cloud infrastructure.

● Architected and implemented a canonical dataset to give the full view of consumption on Spotify across all devices, which empowered teams with a unified and comprehensive understanding of user behaviour, facilitating more informed decision-making and strategic planning.

● Technologies: Apache Beam, Scio, GCP, BigQuery, Scala, Python, Luigi, Data Management, Data Strategy Big Data and Spark Trainer Various Companies Hybrid 2015 – 2023 Instructor in online education platforms upGrad, Simplilearn and AcadGild, to provide industry-relevant programs and training, so that professionals and freshers can develop new, deployable skills in data engineering. Prepared study materials and conducted the live sessions.

● Delivered data training to hundreds of industry professionals.

● Reached more than 500,000 learners on YouTube with original content dedicated to Python

● Created a top-rated course (4.5+ rating) for in-depth, hands-on driven exposure to the features and concepts of Spark Core with tips on tuning its performance. Available at Udemy: Apache Spark Core and Structured Streaming In-Depth

● Technologies: Hadoop, Hive, Oozie, Airflow, AWS, Pig, Sqoop, Flume, HBase, Spark, Kafka, Advanced MapReduce

Senior Data Engineer Careem Dubai, UAE 2018 – 2020 Member of Technical Staff responsible for development of a customised, scalable and modular data platform to enable batch and real time access from diverse systems.

● Developed platform products and capabilities to migrate away from the managed solutions (like New Relic, AWS Kinesis, AWS Glue, AWS Athena), leading to cost savings of over $100K per month.

● Optimised spark applications and in-house ETL solution to reduce run-time and cost by over 30%.

● Reduced data availability from 1 day to 15 mins by spearheading real-time processing efforts and seamlessly integrating Delta Lake, resulting in the establishment of a cutting-edge Near Real-Time Data Warehouse

(DWH).

● Built a data platform from scratch to enable collection, transformation and compliant access to company’s data.

● Technologies: Spark, Kafka, Elastic Search, Hive, HBase, Presto, Java, Scala, Python, AWS, Docker, Jenkins, CDC

Data Platform Engineer Rakuten Tokyo, Japan 2017 – 2018 Developer in the data platform team aimed at adopting a scalable and modern tech stack. Managed stakeholders’ requirements with the best possible strategy.

● Implemented a modern data platform from scratch.

● Revamped the existing data ingestion, transformation, access, cataloguing, governance, documentation and observability.

● Technologies: Python, NiFi, Java, Spark, Kafka, Hive, HBase, Phoenix, Atlas, AtScale Senior Systems Engineer Infosys Bhubaneshwar, India 2012 – 2017 Consulted various clients on adoption of open-source technologies to work with data at scale.

● Migrated analytical workloads from proprietary tools like Teradata to the on-premise open-source Hadoop ecosystem leading to scalability at a fraction of the original cost.

● Implemented and customised solutions related to Map-Reduce, Hive, Oozie and Autosys.

● Developed real time streaming application using Spark Streaming, Kafka, HBase and Hive.

● Awarded with an MFG Rising Star in 2015 for exceeding customers’ expectations and delivering high-quality projects.

● Technologies: Spark, Kafka, HBase, Shell Scripting, Python, MongoDB, Pig, Java, Quartz scheduler, Hive, Tez, Python, Shell Scripting, S3

ADDITIONAL ACTIVITIES

On campus data workshop speaker DRIEMS Cuttack, India 2016 Mentoring young talents and interns Infosys Bhubaneshwar, India 2015 Core coordinator of technical challenges Kurukshetra University India 2010 EDUCATION

Bachelors of Technology (8.3 GPA) Kurukshetra University India 2008-2012 CERTIFICATIONS

• Generative AI with LLM Coursera 2023

• Deep Learning Specialisation Coursera 2018

• Machine Learning Coursera 2018

• Hortonworks Certified Hadoop Component Developer Hortonworks 2016

• Certified in HTML5 and CSS3

REFERENCES available on request

Contact this candidate