Product Manager Evaluation & Data (EC AI)

Job description

Job Description

Rakuten Group, Inc. is a global leader in internet services and has a diverse ecosystem spanning across e-commerce, fintech, communications and more serving approximately 1.8 billion members worldwide. Founded in Tokyo in 1997, the Group operates in over 30 countries and regions with more than 30,000 employees. Based in Singapore's Central Business District, Rakuten Asia Pte. Ltd. serves as the regional headquarters for Asia, driving value through areas such as advertising product development, product strategy, and data management to support Rakuten Group's global ecosystem.

Learn more at

https://global.rakuten.com/corp/ As the Product Manager – Evaluation and Data, your mission is to make the quality of Rakuten's AI products measurable, repeatable, and actionable. You will own the evaluation strategy, process, and platform that enable teams to define success, prevent regressions, make confident release decisions, and continuously improve AI experiences through reliable data and insights.

High Impact Role

You'll establish how AI product quality is measured, directly influencing release decisions, user experience, and business growth.

Evaluation-Driven Culture

You'll turn product requirements and real-world failures into reliable evaluations with talented engineers, data scientists, and domain experts.

Ownership & Autonomy

You'll own the evaluation strategy and platform roadmap, making critical decisions on processes, capabilities, quality standards, and priorities across Rakuten's AI products.

Collaborative Environment

You'll work closely with product, engineering, data science, analytics, and business teams to define success and achieve shared goals.

Position Details Evaluation Strategy & Criteria

Define success for Rakuten's AI products and translate requirements, user needs, and risks into measurable criteria.

Evaluation Quality

Select grading methods, calibrate automated graders with human judgment, and review transcripts and failures.

Platform Strategy

Define the platform vision, roadmap, and requirements with engineering and data teams.

Platform Capabilities

Prioritize dataset management, model/prompt comparison, automated and human grading, multi-turn evaluation, trace review, dashboards, and release integration.

Continuous Improvement

Maintain balanced suites, convert production issues into regression tests, and improve evaluation coverage and accuracy.

Adoption & Decisions

Establish governance, enable team and expert contributions, and connect evaluations with production monitoring, A/B testing, feedback, and business KPIs to guide priorities and releases.

Mandatory Qualifications

Experience: 5 + years of product management experience in AI/ML, evaluation, data products, or platforms. Experience with consumer AI products is preferred.

Data Proficiency

Excellent analytical skills and experience working with large datasets.

Platform & Technical Acumen

Ability to define platform requirements and communicate model/prompt versioning, tracing, data pipelines, and release integration.

Experimentation & Quality

Experience with A/B testing, production monitoring, user feedback, and human evaluation.

Communication Skills

Excellent written and verbal skills, with the ability to present complex evaluation results clearly and concisely.

Collaboration Skills

Proven ability to lead cross-functional product, engineering, data science, and domain teams. Rakuten is an equal opportunities employer and welcomes applications regardless of sex, marital status, ethnic origin, sexual orientation, religious belief, or age.