NeMo turns an agent’s mistaken action into a regression test
NVIDIA’s agent-testing tutorial starts with a concrete failure: an assistant asked to draft project updates also sends the message. NeMo Compass links that behavior to recorded traces, while Eval Author creates tests checking whether sending happened without confirmation. The workflow then changes the agent’s instructions and reruns development and held-out cases, treating the surrounding harness as something to improve and measure.
NVIDIA Technical Blog · Nov 4, 2026
Why it matters
A useful evaluation should check consequential tool actions, not just whether an assistant wrote a convincing final answer.
More context
The example uses fictional-company traces with Hermes in an OpenShell sandbox. Its documented setup requires native Linux Docker, OpenShell, Harbor, an NVIDIA API key and a supported coding agent. Try the supplied example before adapting it; API and hosting costs depend on your setup.
Keep in mind
The demonstration has only two development and two held-out tasks, each run three times. Passing those cases does not establish production reliability or replace a broader regression suite.
Original read