Global LLM Data - LLM Training Operation Coding Analyst

About the team LLM Global Data is a team focused on producing international data for LLMs.

For the training of large models, data is the lifeline of model quality - and the Global Data team is working closely with technical, product, and operations teams to ensure effective data production strategies and execution management.

As a key member of our LLM Global Data Team, the LLM Training Operations Analyst will play a pivotal role in managing the intricate processes involved in training large language models (LLMs) with diverse coding datasets.

This role focuses on overseeing and improving operational workflows, primarily for code-related projects, ensuring they are delivered with high quality and efficiency.

Key Responsibilities and Duties: * Project Management: Lead and manage multiple coding-focused LLM training projects, ensuring timelines, quality standards, and objectives are met.

Track project progress, identify risks, and implement corrective actions as necessary to keep projects on course.

Build and maintain strong relationships with product managers, engineers, researchers, data annotators, and other cross-functional team members.

Communicate project updates, address concerns, and align expectations to ensure successful project outcomes.

Coordinate meetings and discussions with global teams to ensure seamless project execution and work with external vendors and trainers per project demands.

* Workflow Design and Management: Design, manage, and optimize workflows for coding-focused LLM training projects, including training design, QA processes, and performance tracking to meet project needs.

Collaborate closely with product managers, engineers, and cross-functional teams to ensure alignment on quality metrics and project expectations.

* Operational Improvement: Conduct quality and productivity improvement experiments to enhance operational processes for code-related training data.

Lead and support general annotation operation improvement initiatives across various data domains.

Develop and maintain technical guidelines and casebooks to support consistent, high-quality data production.

* Data Checking and Analysis: Design and implement data analysis strategies for LLM coding projects.

Analyze annotation quality, model performance, and dataset coverage using statistical and programmatic methods.

Identify data gaps and failure patterns through slice-based evaluations and error analysis.

Use Python (Pandas, NumPy, Matplotlib) and SQL to generate insights and support model training operations.

Collaborate with researchers to inform training strategies and data improvements.

* Team Leadership and Collaboration: Provide mentorship and guidance to team members, helping to develop their skills and ensuring the delivery of high-quality outputs.

Foster a collaborative environment where team members can share knowledge and best practices to improve overall performance.