Data Engineer Solutions

Bharadwaj N

Data Engineer

Boston, MA +1-857-***-**** ************@*****.***

Summary

• 5+ years of experience in Data Engineering, Data Warehousing, and Analytics with a strong focus on building scalable ETL/ELT pipelines, real-time data ingestion, and cloud-native data solutions using AWS and Azure.

• Proficient in implementing Big Data solutions using Spark, Hive, Kafka, Hadoop, and handling structured, semi-structured, and unstructured data.

• Expertise in processing and modeling large-scale datasets using PySpark, SQL, and Python with deep understanding of Delta Lake, Snowflake, and data governance.

• Strong in DevOps pipeline creation using Docker, Terraform, YAML, and CI/CD implementation in Azure DevOps and AWS CodePipeline.

• Skilled in RDBMS and NoSQL databases including SQL Server, MySQL, Oracle, Cassandra, MongoDB, and DynamoDB with proven performance tuning and optimization.

• Adept at working in Agile/Scrum teams and collaborating with cross-functional stakeholders for delivering analytics-ready and production-grade solutions.

Technical Skills & Tools

Programming languages Python, Sql, Scala, Java, Unix, Shell scripting,T-sql Big data technologies Hadoop, Apache spark, Hive, Pig, Sqoop, Flume, Spark core, Spark sql, MapReduce, Spark streaming, Apache Kafka, Apache Flink, Yarn, Zookeeper, Apache Airflow Frameworks/Libraries Django, Fastapi, Numpy, Pandas, Pyspark, Sqlalchemy, Confluent Kafka ETL Azure data factory, AWS Glue, SSIS, Matillion

Azure stack Azure Data Lake Storage, Azure functions, HDInsight, Databricks, Stream analytics, Event hubs, Logic apps,Virtual machines, Data Explorer AWS stack AWS Athena, S3, Lambda, SNS, EC2, Kinesis, RDS, EMR, IAM, Data pipeline, Route53, Quicksight, Cloudwatch, ECS, Stepfunctions,Elastic search Databases/Datawarehouse Mysql, Postgresql, MongoDB, DB2, Oracle,Sqlserver, Dynamodb, CosmosDB, Azure sql, Cassandra, Azure synapse analytics, Azure data lake gen2, Snowflake, AWS redshift Data Modeling Kimball, Inmon, Data Vault 2.0, SCD type 1, 2 and 3, draw.io. CI-CD/Tools Docker, Azure Devops, AWS codepipeline, Azure Kubernetes, Jenkins, Terraform Version control tools Git, GitHub, Gitlab, Jira

Experience

Rainbird, USA Jan 2025 – Current Senior Data Engineer

• Migrated Hive metastore to Unity Catalog in Databricks for centralized governance and implemented Auto Loader + Delta Live Tables for near real-time processing.

• Orchestrated ETL/ELT pipelines using Azure Data Factory, transforming data via Databricks notebooks and ingesting into Delta Lake and Azure SQL/ADLS.

• Designed dimensional models using Kimball methodology for Power BI executive dashboards, improving performance and usability.

• Developed PySpark notebooks to process 50GB+ data daily, integrated with Power BI dashboards for executive reporting.

• Created serverless pipeline monitoring with Azure Functions and Logic Apps integrated with Azure Monitor, cutting incident resolution time by 30%.

• Scheduled DAGs using Apache Airflow to manage Python-based transformation and integrate with ADF, achieving 99% SLA adherence.

• Built and deployed REST APIs using FastAPI, SQLAlchemy, Pandas; hosted on Azure VM using Azure DevOps and Kubernetes.

• Processed 10GB+ data daily into Snowflake using Snow pipe and DBT scripts for transformation. Polaris, USA Feb 2023 – DEC 2024 Data Engineer

• Managed Agile sprints using Azure Boards and delivered solutions for real-time data ingestion from Event Hubs into Delta Lake with Delta Live Tables for Power BI.

• Developed Spark applications using PySpark and Spark SQL to process data from varied formats (JSON, Parquet, Avro, CSV, XML), improving performance by 30%.

• Automated CI/CD deployment of Synapse, Databricks, and Azure SQL using Docker and Azure DevOps pipelines.

• Wrote and optimized T-SQL stored procedures, triggers, and functions, resulting in a 30% performance boost.

• Migrated legacy SQL Server, Oracle, and MySQL data to Azure Synapse and Data Lake, reducing latency by 30%.

• Collaborated with cross-functional teams to validate and ensure data freshness for reporting and analytics workloads.

• Implemented a Data Vault 2.0 model for customer marketing data to enable traceability, audit readiness, and modular data integration.

Sigma Infosolutions, India Jun 2018 – Dec 2021 Data Engineer

• Built real-time data pipelines with Spark Streaming by ingesting data from Kafka and processing into Cassandra via Kinesis, improving processing efficiency by 20%.

• Created REST APIs using AWS API Gateway and Lambda for orchestrating business logic; outputs routed via Step Functions into DynamoDB.

• Built microservices using Django, integrated with React via REST APIs, and implemented Kafka messaging to reduce response latency by 30%.

• Designed and deployed ETL pipelines in AWS Glue with transformations processed in EMR and loaded into S3, Redshift, and RDS.

• Configured and optimized Hadoop, Hive, Spark, and HBase on EMR clusters to enhance pipeline speed by 20%.

• Actively participated in system testing, job monitoring, log debugging, and deployment scripting.

• Contributed to dimensional modeling efforts by collaborating on source-to-target mappings and applying Kimball principles for Redshift data marts.

• Assisted in preparing normalized (Inmon-style) staging layers for semi-structured data before transformation into business-friendly dimensions.

Magna Infotech, India May 2017 – Apr 2018 Associate Database Support Engineer

• Developed interactive Power BI dashboards and reports to analyze operational performance and provide real- time insights.

• Created Power BI paginated reports for in-depth financial reporting and audit compliance.

• Integrated Power BI with Azure Synapse Analytics to enhance data analysis capabilities and optimize performance.

• Designed and implemented SSIS packages to automate data ingestion, transformation, and loading processes.

• Built efficient SQL-based reporting solutions to enable faster and more accurate data retrieval.

• Implemented data governance and security measures to ensure compliance with industry standards and best practices.

Certifications

• Microsoft Certified: Azure Data Engineer Associate (DP203)

• Microsoft Certified: Azure Data Fundamentals (DP900).

• Snowflake Certified SnowPro Core (COF-C02).

Education

Masters of Computer Science

University of Massachusetts Boston

Contact this candidate