Ruixin Li

AI agents are being deployed commercially faster than evaluation methods have adapted to production realities. Existing benchmarks commonly use retrospectively curated tasks with explicit requirements and deterministic metrics, whereas production work contains implicit constraints, heterogeneous documents, fragmented information, undeclared domain knowledge, and expert-defined success criteria. We present AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies across six O*NET occupational domains. Unlike model-centric benchmarks, AlphaEval evaluates complete agent products—Claude Code, Cursor, Codex, and GitHub Copilot—across 14 scaffold–model configurations. Its evaluation combines reference metrics, formal verification, fixed expert-authored rubrics applied by an LLM judge, and automated UI testing. Beyond the benchmark, we contribute a two-layer requirement-to-benchmark framework: a structured, human-led playbook for eliciting and validating authentic requirements with partner experts, followed by a technical pipeline that turns a specified task package into an automated, reproducible evaluation. The best configuration scores 64.41/100, with large variation across domains and scaffolds.