Senior Applied Scientist, Frontier AI Assets - Assessments (FAI)

Amazon's Artificial General Intelligence (AGI) organization builds frontier models and the AI agents on top of them, and every one of them depends on data we can trust. The Frontier AI (FAI) Assessment team owns the science of dataset quality — for the data that trains frontier models, and for the benchmarks that determine whether a model or an agent actually works.

In this role you will build the automated systems that make quality assessment at scale. Large language model (LLM)-as-a-Judge is the starting point, and our goal is to develop an agentic system that plan its own audits, critiques its own judgments, and improves its own accuracy over time. You will design it, calibrate it against expert human judgment, and set the technical direction for how the organization measures data quality.