Data Engineer Azure

Rishikesh S

Data Engineer

Chicago, IL +1-779-***-**** ***************@*****.***

SUMMARY

• 4+ years of experience in designing, building, and managing cloud-native data pipelines on AWS and Azure.

• Expertise in real-time and batch data processing with Apache Spark (SQL, Streaming), Kafka, Databricks, and Airflow.

• Proven ability to orchestrate ETL/ELT workflows using Azure Data Factory, AWS Glue, and dbt with Snowflake for modern data stack implementations.

• Strong in Python development, including REST API creation using FastAPI, SQLAlchemy, and pandas.

• Hands-on experience with containerization (Docker) and orchestration (AKS/ECS), following CI/CD best practices using Azure DevOps and AWS CodePipeline.

• Deep understanding of data lake architecture, data warehousing, governance (Unity Catalog, IAM), and cost- optimized data storage across Redshift, Synapse, and Snowflake.

• Agile contributor with strong cross-team communication, sprint ownership, and a focus on delivering business-driven data solutions.

• Experienced in integrating Power BI with Snowflake to build optimized datasets, semantic models, and enterprise dashboards with performance tuning using DAX.

SKILLS

Category Skills

Programming Languages Python, SQL, Scala, Java, T-SQL, Shell Scripting, Unix Big Data Technologies Apache Spark (Core, SQL, Streaming), Hadoop, Hive, Pig, Sqoop, Flume, MapReduce, Apache Kafka, Apache Flink, Apache Airflow, Yarn, Zookeeper Frameworks & Libraries PySpark, FastAPI, Django, Pandas, NumPy, SQLAlchemy, Confluent Kafka ETL & Orchestration Tools Azure Data Factory, AWS Glue, Matillion, SSIS, Apache Airflow, DBT, Step Functions, Azure Logic Apps

Azure Cloud Services Azure Databricks, Azure Data Lake Gen2, Azure SQL Database, Synapse Analytics, Event Hubs, Stream Analytics, Azure Functions, Azure VMs, Azure Monitor AWS Cloud Services S3, Lambda, EC2, Kinesis, RDS, Redshift, Athena, EMR, Glue, DynamoDB, Step Functions, Route 53, CloudWatch, ECS, API Gateway, SNS, Elasticsearch, IAM BI Tools Power BI (Reports, Dataflows, Semantic Models, DAX), Tableau Data Warehouses &

Databases

Snowflake, Redshift, Azure Synapse, Azure SQL, MySQL, PostgreSQL, Oracle, MongoDB, Cassandra, DynamoDB, Cosmos DB, SQL Server CI/CD & DevOps Tools Docker, Azure DevOps, AWS CodePipeline, Jenkins, Terraform, Kubernetes (AKS, ECS)

Version Control & Agile Git, GitHub, GitLab, Jira, Azure Boards Operating Systems Windows, Linux

EXPERIENCE

Cigna Healthcare, US Jan 2024 – Current Senior Data Engineer

• Built serverless data pipelines using AWS Glue, Lambda, Step Functions, and S3, handling 10GB+ daily ingestion and transformation.

• Developed real-time streaming pipelines with Kafka and Spark Streaming on EMR, loading data into Redshift and Cassandra.

• Designed and deployed FastAPI-based microservices using API Gateway and Lambda, with data stored in DynamoDB.

• Automated infrastructure provisioning with Terraform, and CI/CD deployments via AWS CodePipeline and CloudFormation.

• Enforced IAM policies, KMS encryption, and Secrets Manager for secure data pipeline operations.

• Implemented observability using CloudWatch dashboards, custom metrics, and SNS alerts for pipeline health monitoring.

• Optimized Spark jobs using partitioning, broadcast joins, and caching, improving performance by 30%. Wipro, India Jan 2022 – Dec 2022 Data Engineer I

• Orchestrated ETL/ELT pipelines using Azure Data Factory, with complex transformations in Azure Databricks (PySpark).

• Handled 50GB+ of daily ingestion using Auto Loader + Delta Live Tables, governed through Unity Catalog.

• Developed event-driven data flows using Azure Event Hubs and Stream Analytics for near-real-time processing.

• Contributed to a parallel analytics stream where dbt was used to model curated campaign data in Snowflake using SQL-based transformation logic.

• Used Azure Data Factory to move Delta outputs from ADLS Gen2 into Snowflake staging tables for analytics benchmarking.

• Collaborated with BI developers and business analysts to translate reporting requirements into scalable dbt and Power BI models.

• Built and deployed REST APIs using FastAPI on Azure Kubernetes Service (AKS), monitored via Azure Monitor and Log Analytics.

• Connected Power BI to Snowflake via ADLS staging layers, building semantic models and dataflows to enable self-service analytics across campaigns and ops.

• Designed performant Power BI dashboards using DAX, custom measures, and incremental refresh, enabling executive-level visibility into campaign KPIs.

Aktrix, India Apr 2020 –Dec 2021 Data Engineer

• Created RESTful APIs by leveraging AWS services (API Gateway, Lambda, Route 53, IAM, S3, CloudWatch) and load the StepFunctions outputs to DynamoDB.

• Built modular ETL pipelines using AWS Glue and Spark on EMR, integrating data from S3, RDS, and Redshift.

• Designed microservices architecture with Django, Docker, and ECS, integrated with Kafka and API Gateway.

• Developed Kafka-to-Cassandra streaming flows using Spark Structured Streaming, enabling low-latency analytics.

• Built event-driven REST APIs using FastAPI + Lambda, with persistence in DynamoDB and Aurora.

• Deployed infrastructure via CloudFormation and managed credentials/secrets through AWS Secrets Manager.

• Implemented full observability using CloudWatch logs, metrics, and alarms, ensuring proactive issue detection.

• Refactored Spark jobs with partition pruning and in-memory caching, reducing job execution time by 40%. Education

Masters in Business Analytics

Lewis University

Bachelors in Information Technology

CVR college of Engineering

Contact this candidate