ReviewBench measures what AI code reviewers catch and miss

GitHub released ReviewBench as a research preview for evaluating AI code-review agents. Its corpus contains 219 public pull requests across 19 languages, with findings gathered from people, models and static analysis. The evaluation separates matching known issues from judging newly discovered findings, and lets readers compare results by severity and tolerance for noisy comments.

GitHub · Oct 5, 2026

Why it matters

Review quality needs both useful findings and low noise; a comment count alone cannot measure it.

More context

Try the public dataset or self-serve runner. Hosted submissions require GitHub sign-in, a container image, configuration and your own model key. GitHub supplies the judge, and leaderboard publication requires maintainer review.

Keep in mind

The grader uses an LLM under a published rubric. Human agreement and production correlation are GitHub’s reported validation, not guarantees for your repository. This is a research preview; 1min has not run an evaluation.

Original read