Big Data Architect

Role: Big Data Architect

Location: Pittsburgh, PA (Onsite)

Duration : Long Term

Job Description

SKill: Big Data Architect, Cloudera Big Data Tool stack (Hive,Impala, OoZie), Pyspark, Teradata, Shell Script, Unix programing, Hadoop

8+ years of data engineering/architecture experience in Waterfall and Agile Methodology, with a proven track record of delivering complex data solutions, preferably in the financial sector.

Proficiency in Python for advanced data analysis, scripting, automation, and integration with Big Data platforms and ecosystems.

Strong SQL experience (Oracle, Hive, Impala) in designing and implementing complex data models, DDLs, and DMLs, with a focus on performance optimization and scalability.

Proficiency in shell scripting for automation, task orchestration, and system integration in Unix/Linux environments.

Expertise in Cloudera Big Data Technologies, including Cloudera Distribution of Hadoop (CDP), Apache Hadoop, Apache Spark, Apache Hive, Apache Impala, and Apache HBase, with hands-on experience in deploying, configuring, and optimizing these technologies for large-scale data processing and analytics.

In-depth knowledge of data governance policies, metadata management, and regulatory compliance requirements, with a focus on protecting sensitive data and ensuring data privacy and security.

Proven experience migrating data from relational databases to Hadoop platforms, with a deep understanding of data migration strategies, tools, and best practices.

Ability to integrate complex big data infrastructure with diverse systems, applications, and data sources, using industry-standard APIs, connectors, and integration patterns.

Familiarity with data encryption tools and techniques, including Protegrity, for securing data at rest and in transit, and ensuring compliance with industry regulations and data protection standards.

Experience working with development tools like Jenkins for continuous integration, deployment, and pipeline automation, streamlining the software development lifecycle and ensuring rapid delivery of high-quality solutions.

Proficiency in Apache Oozie for workflow management, job scheduling, and orchestration of complex data processing workflows in Hadoop ecosystems, ensuring reliable and efficient execution of data pipelines and batch processing jobs.

Hands-on experience with ELK (Elasticsearch, Logstash, Kibana) stack, including configuration, customization, and management of Kibana dashboards for real-time monitoring and visualization of system metrics and application logs.

Effective communication, presentation, and organizational skills, with the ability to articulate complex technical concepts to diverse audiences and influence stakeholders at all levels of the organization.

Proficiency in Microsoft Visio, Excel, PowerPoint, and Word, with the ability to create compelling visualizations, documentation, and presentations to support decision-making and knowledge sharing.

Demonstrated leadership, analytical, and problem-solving skills, with a track record of driving innovation, fostering collaboration, and delivering results in dynamic, fast-paced environments.

Excellent written and oral communication skills, with the ability to convey complex technical concepts clearly and persuasively to technical and non-technical audiences.

Proven ability to work independently and as part of a high-performing team, with a collaborative and customer-centric approach to problem-solving and decision-making.

Commitment to continuous learning and professional development, with a passion for staying abreast of emerging technologies, trends, and best practices in data management, analytics, and related fields.

Strong critical thinking and decision-making abilities, with the capacity to prioritize competing demands, discern critical from minor issues, and innovate creative solutions to complex challenges.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.