Data Engineer

Varadharajan Soorasangaran

Data Engineer

Email: rajvarathan@gmailcom Cell : 952-***-****

SUMMARY

Data Engineer with 5+ years of building data intensive applications, tackling large scale data challenges with Big Data.

Expert in Data management with wide knowledge of Data governance, architecture,Data warehousing, Business Intelligence and analytics

Highly skilled in end-to-end Implementation of Data Lakes/ Data Warehouse from requirement gathering to implementation with adherence to standards .

Experienced in modeling enterprise data across various business units (Sales/Marketing/Finance/Services etc.).

Expert in Hadoop Ecosystems – HDFS,MAPREDUCE, HIVE, IMPALA, AMBARI, OOZIE, Kafka, Cassandra

Posses sound knowledge of Ingress (Ingestion),Egress (Reading) & Processing data in Hadoop cluster using both on-premise & cloud big data tools

Advanced knowledge on Unix Scripting & Python for data processing and file movement and manipulations

Posses sound knowledge and working of SOAP/ REST API’s framework and methods to interact with web services

Proficient in Data Management fundamentals as cited in DMBOK.

Expert in BI Architecture,Design, Development including ETL & Reporting frameworks.

Expertise in Modeling (Conceptual, Logical & Physical) and data security provisioning.

Strong expertise in SQL queries, Views, Store procedures and performance tuning

Proficient in using wide range of BI Tools for design and development of applications.

Proficient in developing challenging dashboards and visualizations (Cross tabs/Matrix, Maps, Bubble/Waterfall/ Key Influencers) and Scorecards

Proficient in technical and business writing, including Architecture diagrams, Process/Data flow, Data Lineage & Data Catalogues, Best practices and WIKI training manuals.

Experience in creating Dashboard reports and automating report distribution and delivery using jobs.

Experienced in all phases of SDLC (Software Development Life Cycle), including requirements gathering, analysis, design, development, implementation, testing and deployment as well as software engineering methodologies like RUP (Rational Unified Process) and Agile methodologies.

Experience in OLTP/OLAP System Study, Analysis and ER Modeling, as well as maintaining Database Schemas like Star and Snowflake Schema(s) used in relational and dimensional modeling.

Expertise in designing and developing Test Plans and Test Scripts for Unit, System and Integration scenarios.

Team oriented with exceptional analytical and problem-solving skills, strong verbal and written communication skills with an excellent interpersonal, planning and coordination skills.

TECHNICAL SKILLS

Data Modeling:

Erwin Data Modeler

SAP Power Designer

Big Data Ecosystems:

Hadoop, HDFS, Kafka, Ambari, Oozie, Hue, Hive, Impala

Languages :

Python

Scripting :

Unix

BI :

Power BI, Cognos Analytics,Qlikview

Visualization:

Tableau

ETL :

IBM DataStage 11.3,

Informatica Power Center 9.1

Relation Database(s):

Oracle 12c, Netezza, SQL Server 2017,

No SQL Database:

Cassandra

PROFESSIONAL EXPERIENCE

Cargill, MN Sep 2019 to Current

Data Engineer

Worked closely with Solution & Data architects on the data source requirements and solution approach to move the source (SAP Success factor) data to Hadoop HDFS

Actively involved in designing the data load process from Middleware (Dell Boomi) to Kafka brokers

Design and development of ETL pipeline to load streamsets (from Kafka) to HDFS file storage

Worked closely with Cargill Data Platform in creating DB views (aggregate & detailed) to store the HR data adhearing to Data Privacy regulations (surrogate key in product layer)

Worked closely with the deployment team on raising request, deployment process and documenting the deployment plans

Involved in deployment and distribution of dashboard to stakeholders

Environment: Hadoop Ecosystem (Kafka,Ambari, HDFS, HIVE & Impala),Unix (File movement,scripting),

SAP Success Factors (Source),Power BI (BI Tool), Jira (Team collobaboration), MS Teams (Documentations)

Cantel Medical, MN November 2016 to Aug 2019

Data Engineer

Configured and implemented Flume for efficiently collecting, aggregating and moving large amounts of

data to HDFS.

Performed ETL using Pig, Hive and MapReduce to transform transactional data to de-normalized form.

Configured periodic incremental imports of data from Sql Server into HDFS using Sqoop and used Hive QL to analyze the partitioned and bucketed data

Developed Hive Queries for data analysis by extending it features by writing Custom UDF’s.

Proactively monitor the overall system from the data load perspective and take appropriate actions to improve performance

Worked on large scale Hadoop cluster for distributed data processing analyzing using Hive, and HBase.

Executed Hive queries on Parquet tables to perform data analysis to meet the business specification logic.

Utilized Oozie workflow engine to run multiple Hive jobs

Importing data from SQL Server to HDFS using a python based on the Sqoop framework.

Developed Hive queries for creating foundation tables for staging the data

Prepared mapping Document prior to development phase

Prepared QA & PROD Data Migration document prior to migrating the project code to higher environments.

Performed Unit testing and System testing to validate data loads in the target.

Involved in modeling Sales/Marketing/Tracings data model for Global Sales Data Warehouse

Created and Designed ETL pipeline to load data from staging, curated & product layers

Worked closely with business analysts & solution architect on clarifying requirements and business logic for product layer tables & views

Environment: SAP Power Designer (Data Modeling), Hadoop Ecosystem (PIG,Flume,Hive, HDFS,MapReduce,HBase, Sqoop, Flume) & SQL Server 2017(DB)

Optum, MN August 2015 To September 2016

BI Architect

Design of HEDIS Data Model and ETL Packages based on underlying warehouse data.

Provided design recommendations and support for offshore development teams.

Effectively managed Scrum meetings and follow-up with Product owners on Application discussions.

Documented user stories, assignment of tasks and timelines adhering to Agile methodology.

Actively involved in design and solution discussions with Data and Application Architects.

Environment: Power Designer (Data Modeling), Cognos 10.2 Framework Manager (BI metadata modeling), Data Stage (ETL), Netezza & SQL Server (DB)

Selective Insurance, NJ August 2014 to July 2015

Senior BI Developer

Data modeling of Selective Billing System.

Designed ETL Packages based on underlying warehouse data.

Designed and developed reports based on Accounting, Treasury and Billing service’s needs.

Designed several Active report POC’s with Insurance Billing data.

Documented User access, Security and deployment mechanism for future purposes.

Deployment of reports and management of scheduling with the help of Zena scheduler.

Environment: Erwin Data modeler (Modeling), Cognos 10.2 (BI), Oracle 12c & SQL Server 2012 (DB)

Hess Corporation, NJ July 2013 to July 2014

BI Lead

Designed and modeled Retail data model.

Designed Corporate Scorecard & Projections report for Executive Business leaders, as well as active reports for retail categories which included Beverage, Fast Food.

Designed several Active report POC’s which included Food Categories, Beverage, Fuel from retail.

Designed and deployed event studio reports to provide information of reports that failed to run per schedule.

Devised a process to automate frequent report requests from category managers.

Environment: Power Designer (Modeling), IBM Netezza (DB), IBM Cognos (BI) & MS SSIS (ETL)

Merck Pharmaceuticals, NJ October 2012 to June 2013

BI Lead

Anchored the requirement discussion workshops with business and suggested metrics and reporting framework which were critical to business needs.

Modeled Sales data model for Sales Reporting.

Responsible for designing Materialized views in Oracle for aggregating data from warehouse tables to provide faster retrieval and performance while generating reports.

Actively worked with UI design team to get the most appealing look and feel in iPAD conforming to client specifications.

Designed the framework model with the materialized views and other needed dimension tables.

Architected the active report design and coming up with iPAD specific features such as vertical scroll, slider control, navigation between tabs, Zoom-in feature for charts.

Architected the bursting in reports to send reports to users based on user’s position and distribution of reports to users by Cognos Jobs.

Conducted regular steering committee meeting with Business and Merck IT in coming up with Process for distributing reports to users on monthly basis for various sales force levels.

Facilitated Change Management across entire process from Project conceptualization to Testing through Project Delivery.

Environment: Erwin Data Modeler (Modeling), Cognos 10.1 (Active reports, BI), Oracle 10g (DB), UNIX (Scripting)

Novartis Pharmaceuticals, NJ May 2012 to September 2012

BI Support & Enhancement Lead

Worked on supporting weekly data load for SMART in monitoring the jobs.

Automated the system by scheduling the weekly jobs using Control-M and triggering reports to users with Cognos Event studio.

Architected the conversion of online version of SMART reports to active reports which could be accessed on sales forces iPAD’s.

Worked in modifying the report as well as backend Oracle procedures to meet change in requirements suggested by business.

Conducted regular steering committee meetings with Business and Warehouse support teams in communicating any data load delays and ensured business & IT teams are informed of any impacts well ahead.

Worked on Cognos admin activities like package import/export, deployment of contents to Dev/UAT/Prod servers.

Environment: Cognos 10.1 Active reports, Oracle 10g, Cognos Admin, CTRL-M Scheduler

Pfizer Pharmaceuticals January 2011 to April 2011

BI Developer

Architected Cognos Go Office Report design & Development for Global Talent Planning team.

Extensively involved in Data Quality Scorecard design and development for Pfizer Global HR team(s) using Cognos Report Studio & JavaScript.

Involved in Divestiture solution design and implementation for Pfizer Human Resources team.

Architected the solution design and analysis of implementing the global Prompt option to be used across the reporting and simplified user Interface.

Environment: Cognos 8.4 Report studio, Cognos Go-office, Oracle 10g

JP Morgan Chase, NJ August 2009 to Dec 2010

BI Developer & Data Analyst

Interacted with business groups to understand needs and provide daily reports on the loans from source system.

Worked extensively in creation of Weekly Consolidated Status Report on the YTD loan categories for higher management.

Automated the Excel based reports to Cognos online reports scheduled to be sent to users via e-mail.

Analyzed and worked on Metric Designer for extracting and loading data for Scorecards.

Designed Dashboard report page based on the requirements.

Created calculated data items in Cognos Report Studio based on the reporting requirements.

Created List, Crosstab and Bar/Pie charts for the Dashboards.

Worked extensively in setting up drill through capability (In List, Crosstab and Bar Graphs) for data validation.

Worked on Intensive data collection from various systems with Oracle 10G as the backend.

Designed the data model for the reports that needs to be delivered to business based on the requirements.

Environment: Cognos 8.4 Metric designer, Report studio, Oracle 10G, Excel macros

BNYM, New York, NY June 2009 to August 2009

Audit Report Developer

Actively participated in requirements gathering, design and implementation phases.

Worked extensively on report development using ACL 9.0 for the Trust Accounting system.

Created several different types of ad-hoc (Simple List, Grouped List, Section Heading, Crosstab, and Nested Crosstab) Reports and represented data graphically by Charts utilizing Query Studio. Created several different types of Professional Reports (Mailing Labels, Web Page Style, Cascading, Linked /Drill Through, Production, Business, Ranking, etc.…) with complex formatting utilizing Report Studio.

Creation of standard filters, calculations, prompts and conditions in catalog.

Created List Reports and Cross Tab Reports and Drill Through reports.

Deployed Cubes on the web and provided Drill through capabilities from Enterprise server to IWR.

Created Cube, added User Class views, Dimension Views, Partitioned cube manually, added Drill Through capability, Cube Group using various data sources in Transformer.

Assisted users in troubleshooting their reports.

Developed Standard templates in Report Studio for developing reports.

Environment: ACL 9.0 (Front End), Oracle 10G (DB), MS Access, Flat file (Source Files)

Netapp (Network Appliances Inc, US) August 2007 To April 2009

DW & BI Support Specialist

Interacted with the business on the new enhancement requests that are needed as part of the existing design to the reports/ ETL tables.

Designed new reports, modified existing framework models in Cognos8 to cater to enhancement requests.

Anchored training sessions with business and explained Cognos NGS Reporting model to users for their Ad-hoc reporting.

Actively worked on enhancing existing Cognos FM models based on business request.

Involved in solution design, development and testing of enhancements.

Extensively worked on modification of ETL jobs, PL/SQL procedures to meet the enhancement requests.

Environment: Cognos 8 (BI), Datastage 9.2 (ETL), Oracle 9i (Db), UNIX (Scripting)

Communication Metrics Dashboard – POC June 2007 To August 2007

Data Modeler, Report Developer

Architected the Universe design based on requirements; created calculated measures at the Universe for utilizing in the report layout.

Actively worked on creating Bar/Pie charts and graphical representation of the metrics in the report to provide a consolidated dashboard look.

Environment: BO XI R2 Desktop intelligence / Web intelligence, BO Universe (BI)

MCI Reporting April 2005 to May 2007

BI Functional Analyst / Support Specialist

Worked with business users, functional lead to understand and document the business requirements.

Imported data from database into Framework Manager.

Worked on Framework Manager to create models, packages and publish packages; built models containing namespaces, query subjects and query items from scratch as well as from imported metadata.

Planned and created professional reports in Report Studio and ad hoc reports in Query Studio.

Generated various List Reports, Grouped Reports, Cross tab Reports, Chart Reports and Drill-Through Reports.

Integrated Cognos 8 with third-party security providers LDAP and distributed Reports through Bursting, E-mail and uploading to folders.

Deployed packages, folders and content store from a development environment to test and then to production environments.

Worked on Analysis Studio to develop Cubes; imported Power Cubes to Cognos 8 and Cognos Connection.

Tuned the performance of the Cognos BI 8 for quick runtime of reports.

Trained business users to develop and use reports with Report Studio and Query Studio.

Environment: Cognos 8 (BI), Oracle 10g (DB), PL/SQL (programming)

EDUCATION

Master’s in Network Engineering from Anna University, Tamilnadu, India

Bachelor of Computer Science & Engineering from MK University, Tamilnadu, India

CERTIFICATIONS

IBM Certified BI Report Designer

IBM Certified BI Data Modeler

Oracle Certified Associate

Certified SCRUM Master / Product Owner

Contact this candidate