Data Engineer Power Bi

Diwakar Teja Ravilla Mobile: +1-425-***-****

Email: **.****@*****.*** Linkedin

Staff Data Engineer with 20 years of experience in design, development and implementation of Datawarehouse, Data Lake, Delta Lake, Big Data, Cloud Data Migration, Real time data/batch processing, ClickStream Processing and A/B Testing.

Skills

●Proficient in Python, Java, Scala, GoLang, T-Sql, Postgresql

●AWS Redshift, SQL Server MS, Oracle, Aurora db (postgres/mysql), Teradata, MongoDb, Cassandra

●Hive, EMR, Tez, Presto, Zeppelin

●Elastic Map Reduce, Spark, and Cosmos (Microsoft Map reduce)

●Cloud IAAS services like Terraform

●Tableau, Power BI, Cubes (SSAS), Grafana

●Apache Airflow, Oziee

●File formats – avro, parquet, orc, delta

●Kafka streaming

●Cloud Technologies -- AWS, Azure, GCP

●CI and CD pipelines, Jenkins, VSO, DevOps

●GenAI, Vector databases

Experience

Lead Data Engineer at Microsoft (May 2023 to till date)

●The source data for CELA (Microsoft Legal Team) data warehouse has been migrated from Oracle on-premise to Azure Sql. With this, the entire source data architecture has been changed.

●Re-engineer of existing DWH and understanding existing data model, created source to target with new datasource.

●Designed and implemented end-to-end data flow for the new Datawarehouse using Azure DataFactory.

●Design and develop interactive reports and dashboards in Power BI.

●Devleoped DAX (Data Analysis Expressions) for custom calculations and aggregations. Designed and built SSAS cube, developed all the measures, metrics and KPIs which are used for reporting.

●Developed visualizations and dashboards that effectively communicate trends and insights.

●Manage Power BI Service and ensure proper configuration and access.

●Implement Alert systems, data governance, and security best practices.

●Lead/mentor a team of four developers and front face to all stakeholders.

●Implemented Devops using Azure ADO to automatically check-in/out with different environments.

Staff Data Engineer at Stockx (Sep 2021 to April 2023)

●Design and Implementation of end-to-end data flow for the following complex pipelines like Bids/Asks, Click Stream,A/B Testing, OrderManagement, Product catalog .

●Design and implementation of data migration from Redshift to Databricks Delta lake

●Establish best practice like standards and guidelines for design & development, deployment, support and analytics and mining.

●Developed Spark Applications by using PySpark, Scala and Implemented Apache Spark data processing project to handle data from various Streaming sources (Kafka).

●Design Implemented pipeline for setting AutoWLM in Redshift

●Implemented Infra as a Service using Terraform, so that all the AWS resources are maintained through Terraform.

Data Architect at Sogeti (Oct 2019 to Sept 2021)

●Engineering efforts including daily monitoring batch process/alert notification over failover, Data Governance, Usage, Data Quality etc.,

●Design and Implementation of Event Based ETL Architecture.

●Establish best practice like standards and guidelines for design & development, deployment, support and analytics and mining.

●Worked with Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLlib, Data Frame, Pair RDD's, Spark YARN.

●70% Technical hands on and 30 % People Management/Performance Management.

●Design and develop interactive reports and dashboards in Power BI.

●Devleoped DAX (Data Analysis Expressions) for custom calculations and aggregations. Designed and built SSAS cube, developed all the measures, metrics and KPIs which are used for reporting.

●Developed visualizations and dashboards that effectively communicate trends and insights.

●Manage Power BI Service and ensure proper configuration and access.

Senior Data Engineer at Coupang (Oct 2016 to Oct 2019):

●Build top priority tables for Finance, Catalog, Fulfillment Center and retail.

●Designed and developed a multi peta byte dimension table on catalog which is used across entire company.

●Migrated entire data marts from On-premise db to Cloud MPP db (redshift)

●Designed and build usage data pipeline on redshift to know, exact usage on tables and columns by the end-users

●Designed and developed automation testing frameworks to validate daily pipeline

●Developed Early SLA Alert Notifications which helps to understand any events causing delay of daily batch process, and thus alerting the On-call Engineers to take necessary action

●Designed and implemented Security on Data Lake for various groups using Apache Ranger.

Senior Software Engineer at Microsoft (as Vendor) (Apr 2013 — Oct 2016):

●Designed and Development of DataMart in SQL Azure 2016

●Created End to End flow of Data Pipelines (Ingest, Prepare, Transform and Publish concepts).

●Design and implementation framework of Linked Services, Datasets, Logging Orchestration of Data Pipelines in Azure Data Factory

●Designed and built all the Power BI dashboards.

●Designed and built SSAS cube, developed all the measures, metrics and KPIs which are used for reporting.

●Developed Complex Power shell scripts to handle the data orchestration in daily E2E pipeline (Cosmos Scripts submission, Checking the availability of cosmos streams, Custom Error handling to make the script resilient of environmental issues... etc.)

Technical Consultant at Wipro Technologies Ltd (July 2009 to Apr 2013):

●Designed and implemented the data warehouse ETL.

●Developed complex EIP Platform ETL scripts.

●Developed Power BI reports to provide advanced slice & dice functionality to user.

●Created a solution to implement SCD fact tables by giving the ability to view the day wise data at the granular level

●Designed and implemented the data warehouse ETL using SSIS, and then migrated to MIST

●Implemented Row-level Security for users based on Business Rules.

Test Engineer – Infosys Technologies Limited (July 2004 to July 2009):

Designed the ETL Architecture based on Business requirement.

Implemented Data modeling using ERWIN tool

Developed Complex stored procedures including slowly changing dimension implementation.

Strong knowledge in performance tuning of both SQL and cube.

Involved in Project Design Documentation

Developed ETL framework using Informatica 8.1

Implemented Shell scripts for file processing and Archival processing

Education:

●Bachelor of Technology (Electrical & Electronics Engineering) from Jawaharlal Nehru Technological University, Hyderabad, India (1998 – 2002)

Contact this candidate