Measuring an agent’s framework cost takes repeated runs

Nolan Lawson asked Opus 5.5 to build the same Pokémon application across 12 JavaScript frameworks, then checked functionality, accessibility and performance. Repeated runs produced substantial variation in estimated model cost and output. The experiment makes a useful comparison method visible, while its single application and model cannot establish a universal framework ranking.

Nolan Lawson · Oct 28, 2026

Why it matters

Framework choices for agent-assisted work need quality checks alongside token or cost estimates.

More context

Most frameworks had five runs; Angular had three successful runs after two failures. Reported dollar figures are API-equivalent estimates from Claude Code, not the author’s actual per-run bill: he used a subscription. Agents received a functionality test suite but not the Lighthouse evaluation criteria.

Keep in mind

Random model behavior, framework-specific tooling and variable Lighthouse results affect the comparison. Lines of code are only a rough maintainability measure.

Original read