Big Data Engineer

Title: Big Data Engineer

Location: Columbus, OH, United States

Length: Long term

Restriction: W2 or C2C

Description:

*** Webcam interview *** Initial PO till 6/2021 with extensions; usually project goes for 3+ years with this customer *** Initial remote due to covid then Onsite; ***

6+ years of experience with Big Data, Hadoop on Data Warehousing or Data Integration projects.

Analysis, Design, development, support and Enhancements of ETL/ELT in data warehouse environment with Cloudera Bigdata Technologies (Hadoop, MapReduce, Sqoop, PySpark, Spark, HDFS, Hive, Impala, StreamSets, Kudu, Oozie, Hue, Kafka, Yarn, Python, Flume, Zookeeper, Sentry, Cloudera Navigator) along with Oracle SQL/PL-SQL, Unix commands and shell scripting;

Strong development experience in creating Sqoop scripts, PySpark programs, HDFS commands, HDFS file formats (Parquet, Avro, ORC etc.), StreamSets pipeline creation, jobs scheduling, hive/impala queries, Unix commands, scripting and shell scripting etc.

Writing Hadoop/Hive/Impala scripts for gathering stats on table post data loads.

Strong SQL experience (Oracle and Hadoop (Hive/Impala etc.)).

Writing complex SQL queries and performed tuning based on the Hadoop/Hive/Impala explain plan results.

Proven ability to write high quality code.

6+ years of experience building data sets and familiarity with PHI and PII data.

Expertise implementing complex ETL/ELT logic.

Develop and enforce strong reconciliation process.

Accountable for ETL/ELT design documentation.

Good knowledge of Big Data, Hadoop, Hive, Impala database, data security and dimensional model design.

Basic knowledge of UNIX/LINUX shell scripting.

Utilize ETL/ELT standards and practices towards establishing and following centralized metadata repository.

Good experience in working with Visio, Excel, PowerPoint, Word, etc.

Effective communication, presentation, & organizational skills.

Familiar with Project Management methodologies like Waterfall and Agile

Ability to establish priorities & follow through on projects, paying close attention to detail with minimal supervision.

Required Education: BS/BA degree or combination of education & experience