Data Engineer Power Bi

Pooja Reddy Gopu

Data Engineer

+1-913-***-**** *****.*@**********.*** USA

SUMMARY

Data Engineer with 4+ years of hands-on experience designing and optimizing scalable data pipelines and analytics solutions. Proficient in SQL, Python, and Azure, with a strong background in building robust ETL/ELT workflows and data models for efficient data integration and transformation. Adept in data warehousing and relational database systems, including schema design using Star and Snowflake models. Experienced in leveraging Apache Spark and Azure Databricks to process large datasets and derive actionable insights. Developed dynamic dashboards using Power BI and Tableau, enabling data-driven decision-making through real-time KPI tracking and reporting from Azure Synapse Analytics. SKILLS

Programming Language: Python, R Programming, Scala, SQL, NoSQL Big Data Ecosystem: Apache Spark, Apache Kafka, Apache Nessie, Hadoop, Hive, HDFS, Zookeeper Cloud: AWS (EC2, S3, RDS, Lambda, Glue, AWS Pipeline, Redshift, DynamoDB), Azure (Databricks, Synapse Analytics, Data Factory) Visualizations: Tableau, Power BI, Excel, Alteryx

Packages: NumPy, Pandas, Matplotlib, Seaborn, PySpark, Scikit-learn, TensorFlow ETL/ELT Tools: Fivetran, SSIS, Informatica, Data build Tool (DBT), Apache Airflow, Talend, Airbyte, Databricks Database & Tools: GitHub, Git, GitLab, SQL Server, PostgreSQL, Terraform, Cassandra, MySQL, Snowflake EXPERIENCE

Data Engineer, Blue Cross Blue Shield, USA Jul 2024 – Current

Deployed automated data pipelines with Apache Airflow and Azure Data Factory to schedule, orchestrate, and monitor data workflows, reducing manual tasks by 40% and boosting data accuracy and pipeline efficiency.

Established and automated secure data pipelines on Azure (Data Factory, Azure Databricks, Logic Apps) to ingest, transform, and store over 2 TB of daily financial transaction data, ensuring real-time availability for analytics and regulatory compliance.

Engineered an efficient Azure Databricks framework in collaboration with cross-functional teams, enhancing structured and unstructured data processing by 30% through optimized cluster configurations, caching strategies, and Spark optimizations.

Built scalable data lakes on Azure Data Lake Storage and implemented Azure Synapse Analytics for data warehousing, managing over 100TB of structured, semi-structured, and unstructured data to support big data processing for analytics, reporting, and machine learning.

Implemented incremental data models with Azure Synapse pipelines and PySpark on Databricks for large financial datasets, boosting pipeline efficiency, cutting cloud costs by 20%, and improving reporting accuracy across analytics workflows.

Improved ETL workflows through testing and debugging of SQL scripts and Python code in Azure Databricks and Synapse, resulting in a 50% increase in data processing efficiency and ensuring seamless data integration with downstream systems. Data Engineer, Informative Web Solutions, INDIA Mar 2020 – Jul 2022

Led the migration of legacy reporting systems to Power BI, partnering with stakeholders to define requirements and enhance data visualization, reducing report generation time by 40% and improving decision-making.

Developed and automated AWS Data Pipeline workflows to streamline data integration from multiple sources, reducing manual intervention by 25% and improving data flow reliability through enhanced error handling and validation processes.

Leveraged Spark SQL and MLlib to perform complex data analysis, generating actionable insights that improved business decision-making and stakeholder engagement.

Built various ETL/ELT pipelines using Databricks notebooks and SQL to extract, transform, and load data from multiple sources into the data warehouse and lakehouse environments.

Integrated Airflow with AWS to monitor multi-stage workflows with tasks running on Amazon SageMaker, and contributed to CI/CD solutions using Git and Jenkins for configuring the big data architecture on AWS.

Collaborated with Data Governance teams to enforce metadata standards and lineage tracking within Talend Data Catalog, improving data traceability and trustworthiness for internal stakeholders and auditors.

Streamlined Hive queries for performance optimization by activating techniques such as partitioning, bucketing, and compression to handle large datasets effectively, implemented data processing systems by MapReduce to handle workloads efficiently and accurately.

Data Engineer, Sage Softtech, INDIA Apr 2019 – Feb 2020

Collaborated with cross-functional teams to define data requirements and ensure alignment with business objectives.

Designed and implemented data warehouse solutions on Snowflake and Redshift, improving analytics performance.

Ensured data compliance by applying RBAC and encryption standards to enforce governance and security.

Conducted data quality assessments and testing to ensure data integrity, reliability, and alignment with business standards.

Developed ETL jobs using PySpark, implementing data lineage to track data transformations through stages, resulting in a 25% reduction in data processing errors.

EDUCATION

Master’s in Computer Science, University of Missouri Kansas City, USA Aug 2022 – May 2024 B.Sc. Computer Science, Loyola Academy Degree& PG College, Hyderabad Jun 2017 – Apr 2020

Contact this candidate