Data Engineer Senior

Naga Swetha Chappidi

***********.**@*****.*** +1-662-***-****

Senior Data Engineer ETL & Cloud Data Pipelines AWS Pentaho 12+ Yrs Experience

Profile Summary

Over 12 years of experience as a Data Engineer, specializing in ETL, Business Intelligence (BI), and AWS cloud technologies.

Skills Summary:

ETL (Pentaho, SSIS & Informatica) AWS (Lambda, Glue, S3, Athena) Python SQL Snowflake Shell Scripting BI Reporting (Jasper, Pentaho BI) Data Warehousing Agile

More than 8 years of hands-on experience with Pentaho Data Integration (PDI), including advanced ETL development, metadata injection, and integration with various data sources.

5 years of experience using AWS services like Lambda, Aurora, EC2, CloudWatch, Glue, S3, and EventBridge for cloud-native data workflows and automation.

Expertise in designing and implementing end-to-end data integration pipelines using Pentaho, SSIS, and Informatica PowerCenter.

Skilled in building interactive BI dashboards and reports using tools like Pentaho BI, Qlik Sense, Jasper Studio, and Yellowfin BI.

Strong background in data modeling, including Star Schema and Snowflake Schema, with experience designing Fact and Dimension tables.

Proficient in Python, SQL, PL/SQL, and UNIX shell scripting to support data transformations, validations, and automation.

Hands-on experience with event-driven ETL workflows and integrating serverless AWS components.

Comfortable working in Agile teams, actively participating in sprint planning, daily standups, and retrospectives.

Experienced in collaborating with globally distributed teams, effectively managing communication and tasks across onsite-offshore models.

Technical Proficiency

ETL Tools: Pentaho Data Integration (Kettle), Informatica PowerCenter

Big Data Tools: Hadoop, Hive, HDFS, MapReduce, Sqoop, MongoDB

Cloud Platforms: AWS (S3, EC2, Glue, Lambda, Athena, EventBridge, SNS, Aurora, Dynamo DB, CloudWatch, Lambda)

Programming/Scripting: Python, SQL, PL/SQL, UNIX Shell Scripting, JavaScript

Databases: Oracle 10g/11g/12c, MS SQL Server, MySQL, PostgreSQL, Aurora, DynamoDB

BI & Reporting Tools: Pentaho BI, Pentaho Report Designer, Jasper Studio, Yellowfin BI

Tools & Utilities: Git, JIRA, Confluence, Splunk, SQL Developer, SQL Workbench, TOAD, SourceTree, Postman, Mountain Duck, S3 Browser, DB Viewer

Operating Systems: Windows, Linux (Ubuntu), Solaris

Education

Master of Computer Applications (MCA) – JNTU Anantapur, 2010

Bachelor of Science (Mathematics, Physics, Chemistry) – S.K University Anantapur, 2007

Professional Experience

Senior Data Engineer

Nallas Corporation / Verisk — Jun 2024 – Present

Created and optimized ETL processes using Pentaho PDI, successfully extracting data from legacy systems and ensuring smooth data flow.

Leveraged metadata injection to streamline and improve Pentaho workflows, boosting performance and scalability.

Integrated REST APIs, JSON, and AWS S3 with Pentaho transformations, enabling seamless data exchange between systems.

Migrated Lambda event triggers to AWS Batch Queue, improving batch processing efficiency and reducing overhead.

Used AWS Lambda and Python to automatically trigger Pentaho jobs on an EC2 instance, streamlining the process.

Integrated SQS queues for event-driven messaging and notifications in data pipelines.

Optimized Aurora Postgres SQL databases for fast and reliable query performance.

Took ownership of Snowflake tables, optimizing them for fast, efficient queries and smooth data storage.

Managed S3 lifecycle policies, automating data archiving and deletion to optimize storage costs and organization.

Created and configured a new staging region for efficient data processing and storage.

Created insightful business reports using Jasper Studio, helping teams make informed decisions based on accurate data.

Managed version control using Git, tracked progress with JIRA, and documented key information in Confluence for team collaboration.

Automated workflows for data processing using EventBridge and AWS Glue.

Technical Environments: Pentaho, AWS, PostgreSQL, SQL Server, DB Viewer, SQLite, GIT, Agile, SnowFlake, Python, Shellscript, Jasper Studio

Lead ETL Developer

SG Analytics / BNY Mellon — Dec 2021 – Dec 2022

Automated existing ETL processes using Pentaho 9.0.

Built complex ETL transformations with meta data injection and Oracle 12c.

Scheduled transformations and jobs, integrating external services via web APIs.

Managed source code versioning using GitLab.

Technical Environments: Pentaho 9.0, Oracle 12C, Oracle SQL Developer, GIT LAB, FTP

Senior Data Engineer

Nallas Corporation / Verisk Analytics — May 2019 – Dec 2021

Project: Loss Cost Extraction

Led the development and maintenance of ETL pipelines, taking charge of everything from estimations and code reviews to ensuring thorough unit testing.

Automated data flows using Pentaho and managed large datasets via AWS S3/FTP.

Wrote and executed AWS CLI commands to automate and manage AWS resources efficiently.

Guided junior developers, led agile sprints, and collaborated cross-functionally.

Identified and resolved data quality issues to ensure data accuracy and consistency.

Supported the QA team in creating test data for various business scenarios.

Technical Environment: Pentaho DI, AWS Architecture, FTP EC2, S3 Buckets, PostgreSQL & Aurora DB

Project: Stat Implementation

Developed and delivered data integration jobs based on client requirements.

Analysed complex business requirements and created ETL code using Pentaho Data Integration.

Worked with complex SQL queries to calculate key formulas for financial reports.

Kanban method and Agile Manifesto for Onsite/Offshore.

Extensive experience in Pentaho Data Integration, administration & AWS Architecture.

The development activities and versioning take place with Atlassian Jira/GIT Hub applications.

Responsible for Development Activities, which includes Design, Estimations, Code review and validating the Unit test cases.

Possess experience in data ingestion pipelines using AWS/Java (Developed a Lambda function in AWS environment using Java to fetch data from different source to write into S3 storage).

Code/ETL Automation expertise with good working knowledge on Java Scripts, Static Web Application development, etc.

Technical Environment: Pentaho DI, Talend DI & AWS Architecture, Athena, S3 Buckets, Aurora DB

Senior Associate

Cognizant Technology Solutions / Comcast — Dec 2017 – Apr 2019

Migrated complex data structures from legacy systems to AMDOCS.

Developed Pentaho ETL pipelines and handled performance-tuned SQL queries.

Managed code migration and integration using tools like WinSCP and Putty.

Understand complex business requirements and develop ETL code in Pentaho Data Integration.

Dealt with complex SQL queries to attain major formulas for finance deliverables.

Responsible for Code Migration

Design and Implemented ETL for data load from SQL Server to Target.

Senior Associate Consultant

Infosys Ltd / Bank of America — Aug 2014 – Dec 2017

Project: NGO GFCC Approval Flows

Automated data extraction from IRS websites and developed alerts/reporting mechanisms.

Designed and executed ETL strategies using SSIS.

Designed, developed, and deployed ETL packages using SSIS to extract, transform, and load data from various sources (SQL Server, Excel, Flat Files, XML, etc.).

Used variables, expressions, and parameters to make SSIS packages dynamic and reusable.

Scheduled and automated package execution using SQL Server Agent and Windows Task Scheduler.

Optimized package performance using lookup caching, block transformations, and parallel execution.

Created and managed configuration files to deploy packages across different environments (Dev, QA, Prod).

Collaborated with DBAs, developers, and business analysts to understand data requirements and ensure accurate data delivery.

Documented ETL processes and data flow diagrams for ongoing support and future enhancements.

Technical Environment: Pentaho Data integration 6.1, SSIS, Oracle 11g, WinSCP, Putty

Project: E-Business Review

Develop and deliver Data Integration jobs based on the business requirement provided by the client.

Understand complex business requirements and develop ETL code in Pentaho Data Integration.

Dealt with complex SQL queries to attain major formulas for finance deliverables.

Responsible for Code Migration

Design and Implemented ETL for data load from SQL Server to Target

Technical Environment: Pentaho 5.5/6.0, Oracle 11g, SQL Developer

Software Engineer

Tech Vedika Software Pvt Ltd — Jan 2011 – Oct 2013

Project: HRM Reports

Design and Development of Business Analysis Interactive Reports & Dashboards on HR Data according to design specifications.

Generated different reports like Employee Headcount based on Grade, Job etc…, timesheet reports,

Leave summary reports etc. for different level people within the organization.

Used Pentaho Report designer to create various reports having drill down functionality by creating Groups in the reports and creating sub-reports within the main reports.

Designed single value as well as multi-value drop down and list type of parameters with cascading prompt in the reports and dashboards.

Worked on Pentaho Ctools to create dashboards. Added parameter and created JNDI connection to database for fetching input data.

Created dashboards with bar, Line, Pie and multi-plot dashboard containing line and bar chart in one graph.

Deployed reports on Pentaho BI Server to give central web access to the users.

Integrating published BI reports into existing PHP-HRM applications and implemented role-based authentication and authorization.

Technical Environment: Pentaho BI Suite 4.5, Kettle 4.3, Pentaho Report Designer 3.9, MySQL 5.5, SQL Server 2008, SQL Developer, Talend

Project: US Analytics Major

Extensively worked with Business Users in gathering requirements and supporting various issues and providing their solution.

Used Pentaho Data Integration/Kettle to design all ETL Processes to extract data from various sources like Database, flat files and external files, cleanse and then load into target data warehouse.

User the Pentaho DB Repository to create Folders, store Transformations and Jobs, Move, Delete, Restore artifacts.

Supporting daily loads and working with business users to handle rejected data.

Wrote UNIX Shell Scripts for file validations and scheduling jobs using PAN, KITCHEN for automatic execution at the specific timings.

Extensively worked with SCD Type-I, Type-II dimensions and data warehousing Change Data Capture (CDC).

Implemented re-usable code using Mapping steps.

Troubleshoot issues with failed ETL Processes.

Technical Environments: Pentaho Kettle 4.3/4.0, MySQL 5.5, MySQL Workbench

Accomplishments

Delivered fully automated ETL jobs migrating data from AWS/Linux to on-prem systems.

Led training and onboarding sessions for new team members.

Successfully delivered multiple automated solutions with minimal post-deployment issues.

Actively contributed to sprint planning and retrospectives to improve delivery processes.

Awards

Best Performer – Tech Vedika Software Solutions, 2012

Best Find of the Year – Nallas Software Solutions, 2020

Contact this candidate