Data Model Quality Analyst

Job Description

Contract: W2, 6- Months

Location: Hybrid Tues–Thurs On-site in San Jose, CA

Pay Range: $75-85/Hour depending on experience

Responsibilities:

Ensure data quality during task development for human-created content used in training Large Language Models (LLMs).

Define detailed data requirements based on model goals and analysis of model loss patterns.

Design high-quality, nuanced instructions for expert content creators with illustrative examples.

Rapidly iterate with content-production vendors to improve data quality in small batch reviews.

Develop benchmark datasets to assess and validate data quality.

Benchmark vendor outputs against competitor models using both quantitative and qualitative methods.

Analyze data from vendor teams and optimize rater pools for quality.

Audit datasets, identify quality issues, and help develop tools to streamline data review processes.

Use experimentation to demonstrate data impact and uncover model blind spots.

Generate regular stakeholder reports on data quality metrics, iteration times, and vendor performance.

Author and maintain best practices documentation and instructional guidelines.

Requirements:

Bachelor’s degree in English, Creative Writing, Linguistics, Science, Computer Science, or a related field.

3+ years of experience in Technical Writing, Quality Assurance, or a related field.

Excellent writing and editing skills with adaptability to varied styles and tones.

Strong analytical and problem-solving skills.

Ability to work independently and collaboratively across technical and non-technical teams.

Preferred Qualifications:

PhD in a relevant field (English, Linguistics, Computer Science, etc.)

5+ years experience in relevant roles with exposure to LLM development or content creation at scale.

Experience with data benchmarking, quality auditing, or content vendor management.

Familiarity with model evaluation techniques and qualitative dataset analysis.

Russell Tobin / Pride Global offers eligible employees comprehensive healthcare coverage (medical, dental, and vision plans), supplemental coverage (accident insurance, critical illness insurance, and hospital indemnity), a 401(k)-retirement savings plan, life & disability insurance, an employee assistance program, identity theft protection, legal support, auto and home insurance, pet insurance, and employee discounts with preferred vendors.

#LI-JJ2

Hybrid remote