Biswajit Pan

Big Data Engineer

Mobile: +1-201-***-**** E-Mail: adbw08@r.postjobfree.com

Professional Summary

Over 8+ years of development experience applications using AWS Cloud, Java/J2EE, Hadoop and Big Data technologies in Banking, Insurance domains.

Proficient in Java, J2EE, Spring, Kafka, Spark, Scala, Python, Hive, Snowflake, Sqoop, REST, PL/SQL, Jenkins, jQuery, GitHub, Snowflake, Shell, Hibernate, Oracle, Ant, Maven, JIRA, Log4J, GIT, HTML, JavaScript, JSON, JMS, XML, JNDI, XSLT.

Software development experience using Big Data technologies to efficiently solve Big Data processing requirement.

Application Development using AWS Suite like S3, EMR, EC2, RDS, Lambda’s, SNS etc.

Good understanding in Hadoop eco system components HDFS, Map Reduce, Yarn, Sqoop, Spark, Spark SQL, Spark Streaming, and Hive for scalability, distributed computing, and high- performance computing.

Experience in using Hive/SparkSql Query Language for data Analytics.

Active contributor in Apache Spark/Hadoop user forums and proficient in developing custom frameworks.

Implemented Spark using Scala and utilizing Spark Core, Spark SQL API for faster processing of data.

Used Spark-SQL to Load various kind of data and create Schema RDD and loaded it into Hive Tables and handled Structured data using Spark SQL.

Developing Spark programs using Scala and python API's to analysis of data

Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.

Collected and aggregated large amounts of data from RDBMS and stored the data into HDFS for analysis.

Generated the datasets and loaded to HADOOP Ecosystem.

Involved in creating Hive tables, loading with data and writing hive queries that will run internally in MapReduce way.

Worked with various HDFS file formats like Avro, parquet File and various compression formats like Snappy, gzip.

Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Scala and Python.

Worked in all phases of SDLC, DevOps and Agile models - Played Scrum Lead and shadow Scrum Master roles.

Expertise in technically leading Migrations to Open Source Technologies

Ability in technically leading Java/J2EE, Spark, Bigdata developers and teams.

5+ years of Onsite experience working directly with clients and handling onsite & offshore teams effectively.

Capable in initiating POCs, Project estimations, Hardware provisioning, release and configuration management

Highly motivated, results oriented and leadership skills with great team ethics.

Good in mentoring novice programmers and evaluating team performance.

Have the ability to take corrective and preventive actions in delivering desired quality.

Very strong analytical and reverse engineering skills part of technology migrations.

Experienced in working with geographically distributed development teams.

Hands on Experience in developing web applications by using Hibernate and Spring Frameworks with Apache Tomcat.

Expertise in developing data models using Hibernate Object relational mapping (ORM) framework and used Hibernate Annotations to reduce time at the configuration level.

Worked with design and development of robust and highly scalable Web Service platforms and API’s based on REST and SOAP for enterprise applications using AXIS, WSDL, UDDI, XML, JAX- WS, JAX-RS, JAXB, JAX-RPC technologies and Jersey frameworks. Technical Skills

Big Data Ecosystems : Spark streaming, Spark, Sqoop, Hive, Flume, Hadoop, MapReduce, HDFS, Pig, Kafka

Web Services : RESTful Web Services & SOAP

Languages : Java/j2ee, Scala, Python, JavaScript, Servlet, PL/SQL, Unix Shell

Web Technologies : HTML, JSP, JSF, CSS, JavaScript, Python, AJAX Server-Side Frameworks : Spring, Hibernate, Struts 2 IDEs : IntelliJ IDEA, Eclipse, RAD

Build Tools : Maven, Ant

Web Servers /App Servers : Apache Tomcat 6.0/7.0, IBM WebSphere 6.0/7.0, JBoss 4.3 Static Code Analysis : Find Bug/Check Style/PMD

Configuration Tool : GIT-Bit bucket, SVN, CVS, SOAPUI Database : Snowflakes, MongoDB, DynamoDB, Cassandra, HDFS, Oracle 11g,

Defects Triage : Quality Center 9.2.

Cloud Environment : AWS

Academic And Certificate Information

Education University

Degree : Bachelor of Technology (Computer

Science) from WBUT University, India

Certifications : - AWS Certified Solutions Architect.

- Big data expert

- Sun Certified Java Programmer.

Employment History

I am currently working in CapitalOne as Data Engineer from 2018.

I previously worked with Liberty Mutual, AAA Insurance. Company Projects Undertaken

Project: Data Express

Details: DataExpress is a framework to build Streaming applications. The tool helps fetching data from Kafka cluster, save it to AWS S3 for audit purpose and later validates and writes it to Snowflake database. To setup the configuration, one can create this configuration file through a self- service portal and pass on this configuration file provide the same during the deployment. Technology: Kafka, Spark Streaming, Scala, AWS, Java, Rest webservice Client: CapitalOne

Tools: EMR, AWS

Role: Developer and Team Lead

Team Size: 10

Duration: Jan, 2018 – Till date

Accountabilities:

Involved Design of the framework

Involved reading data from Kafka using Spark streaming

Writing data in different format in Amazon - S3(Avro, Parquet)

Writing in Snowflake Database

Design Exception handling for this frame work

Design to create auto Incident Request and Amazon SNS

Design Kafka Failover

Validation of the Schema for incoming data.

Title Customer first(CF)

Technology: Spark, Scala, Hive, Sqoop, AWS, Java, Rest webservice Client Liberty Mutual

Team Size 12

Role Design, Analysis, Detail Design & Developer

Period April 2017 – December 2017

Project Description

Customer first(CF) applications comprised of 3 different application which includes Customer first property(CFP), Customer first Auto (CFA) & CF Personal Liability Protection(CP). While individual modules can be loaded independently of each other, every module shares a set of core code which must present to successfully run CF from a development slice. These applications collectively referred as CF and these applications are used by Liberty Mutual sales staff to get insurance quotes, policy and Policy Endorsement etc. All the applications are in production and the business is rolling out new functions to the application state wise.

Team is working in agile methodology and the new functionalities will be described in Epics and assigned to the team.

Role and Responsibilities

Involved in architecture discussions and requirement analysis.

Involved in discussions with business line.

Manual Code Review

Developed the CF related service.

Title AAA Insurance Home Products

Technology: Java, J2EE, Spring, Hibernate, Soap, Rest Client AAA

Team Size 55

Role Design, Analysis, Detail Design & Developer

Period April 2012 – April 2017

Project Description

This project is a product enhancement and customization based on client specifications and requirements for rolling out a system to support foot print and partner club states. The core base service functionality is provided by Exigen product factory runtime and it is based on policy core common requirements. Creation of New custom products on the top of the policy core system by extending baseline services which includes 3 steps process (1) Product Creation with the help of independent loosely coupled components and respective service configurations, (2) Product Testing and (3) Policy Creation.

Role and Responsibilities

Interacting with the Product Managers and stakeholders at AAA NCNU to better understand their business needs and provide good solutions to them on the same.

Understanding the Business requirements and configure, customize the product as per the requirements document.

Preparing custom component lifecycle diagrams, service design documents for each custom components, UML class diagrams and sequence diagrams as per as the client requirements and specifications

Development of components/artifacts that implements Component interface contract, versioning of the components, and creation of the new business services by extending core policy core services provided by Exigen product factory

Responsible for packaging of the components into libraries, adding/removing attributes, changing attribute labels, configuration of component constraints, component deployments

Personal Details

Present City : Plano, Texas

Languages Known : English.

Visa : Approved (I-140 approved)

Contact this candidate