AI Benchmarking Lead, Performance Benchmarking Evaluation

Job description

Join our mission-critical team supporting Seller Assistant, Amazon's Gen-AI powered copilot that helps sellers navigate Amazon's complex ecosystem and grow their businesses. As a AI Benchmarking Lead, you'll play a pivotal role in ensuring the reliability and accuracy of AI model evaluations as we scale from 61% to 90%+ active seller coverage worldwide.

About Seller Assistant

Seller Assistant is a conversational AI copilot that understands the full context of a seller's business. It intelligently orchestrates back end tools to deliver actionable, drilled-down responses and can independently complete complex tasks on behalf of sellers with their permission.

Our Scale and Impact

- Expanded to 2.44MM sellers (45x growth vs. Dec 2024)

- Currently serving 61% of active sellers worldwide across 9 international stores (CN2XX, IN, UK, DE, JP, BR, MX, AE, SA)

- Supporting four languages: English, Chinese, German, and Japanese

- 2026 Goal: Scale to 90%+ active sellers WW with 5 new store launches (France, Italy, Spain, Canada, Australia)

As a AI Benchmarking Lead, you will benchmark Seller Assistant AI models for relevancy, correctness, and completeness.

Your primary responsibilities include: 1)

• Evaluate audits performed by the core auditing team to increase confidence in evaluation metrics, 2)
• Improve audit reliability and consistency through systematic measurement of auditor accuracy,3)
• Conduct targeted calibration to ensure quality standards across the auditing function, 4)
• Enforce quality standards by quality-checking audits and providing actionable feedback to team members, 5)
• Drive continuous improvement in audit processes and methodologies.

- You conduct quality checks on audits performed by the core auditing team.

- You identify rubric gaps and evaluation ambiguities that lead to inconsistent audit outcomes.

- You surface high-confidence product issues earlier by validating and categorizing model failures.

- You serve as point of contact for annotation tasks across ML data process areas, ensuring quality execution and delivery

- You understand dependencies across ML data workflows and articulate customer impact effectively

- You modify existing annotation methods and update SOPs.

- You document SOP changes, secure approval, share knowledge with the team, and audit adoption and execution

- You test new SOPs and tools, providing feedback on quality and improvement recommendations to support onboarding

Key job responsibilities

- You structure data collection, analyse results and share inputs for SOP changes.

- You collate, track, and report progress on key metrics agreed to with respective stakeholders (e.g., Program managers, Applied Scientist) specific to your functional area.

- You identify operational issues related to process and tooling and recommend suggestions to improve key project metrics such as productivity and quality.