Gen AI / ML Application Testing Engineer

We are looking for a detail-oriented engineer with experience in Gen AI / ML application testing, business analysis, and product validation. You will help shape the quality of next-gen AI products through systematic testing, prompt validation, and tool-driven evaluation.

Key Responsibilities

● Design and execute test cases for Gen AI / ML features and user workflows

● Use GenAI-specific tools to validate prompt consistency, hallucination detection, etc.

● Collaborate with product managers to convert requirements into test cases and test data

● Perform exploratory testing, regression, and prompt-based scenario testing

● Write automation scripts to simulate user behavior and backend interactions

● Track and manage issues using QA platforms and agile tools

● Document test plans, test reports, and AI evaluation metrics

Required Skills

● Hands-on testing experience with Gen AI / ML products

● Experience with LLM testing tools like:

● - Promptfoo (prompt testing & evaluation)

● - LangSmith (LangChain tracing & evals)

● - TruLens (feedback tracking for LLMs)

● - Rebuff (security and behavior testing)

● Solid understanding of LLM behavior, hallucinations, prompt design

● Scripting: Python, Shell, or JavaScript

● REST APIs, JSON, YAML

● Familiarity with PyTest, Postman, Selenium, or similar tools

Nice-to-Have Skills

● Experience testing RAG, chatbot, or LLM agent systems

● Familiarity with LangChain, LlamaIndex, or Haystack

● Business analysis experience in AI projects

● Knowledge of AI/ML model evaluation metrics

Education

Bachelor's or Master's in CS, Data Science, AI/ML, or related field