An evaluation is a saved test case with an expected result. You define what should happen, run the eval, and check whether your app still behaves correctly. When you make changes, run your eval suite to catch regressions before they reach users.
The Evaluations dashboard showing a suite of test cases
In the Evaluations dashboard, select the cases you want to include. You can select all, filter by tag, or pick individual cases.
Selecting eval cases to run
2
Execute the eval run
Click Run Evaluations. Chippy executes each case against your current app and records the results.
An eval run in progress
3
Review pass rates and failures
The results view shows your overall pass rate and highlights failures. Click on any failed case to see what went wrong: which tool was called, what parameters were passed, and how the output differed from the expected result.
Eval run results with pass/fail breakdown
4
Fix issues and rerun
For each failure:
Open the failed case details.
Identify whether the issue is in tool descriptions, parameter handling, or tool logic.
The input is ambiguous or phrased in an unexpected way. These test whether your tool descriptions are robust enough to handle natural language variation.
