Eval & Safety
Evaluation Dashboard
Offline quality signals for the Core Case Router and its four specialists. Every metric is measured against a labelled golden set, not production traffic.
2 metrics outside tolerance — tool success rate, false positive rate · geo-mismatch. Autonomy thresholds stay unchanged until the next passing run.
Quality metrics
Intent classification accuracy
On target94.2%
Target ≥ 92.0%
+0.8 pts vs. last run
1,840 labelled contact-center transcripts · 12 intent classes
Tool success rate
Off target98.6%
Target ≥ 99.0%
−0.3 pts vs. last run
Core banking timeouts account for 71% of failures
Override rate
On target11.4%
Target ≤ 15.0%
−1.2 pts vs. last run
Associate changed or rejected a recommended action
False positive rate · geo-mismatch
Off target6.8%
Target ≤ 5.0%
+1.1 pts vs. last run
VPN exit nodes drive most false geo-mismatch flags
Avg. confidence calibration error
On target0.043
Target ≤ 0.050
−0.006 vs. last run
Expected calibration error across 10 confidence bins
Recent offline evaluation runs
| Run | Date | Scenario set | Pass / fail | Result | Duration | Details |
|---|---|---|---|---|---|---|
| EV-2291 | 08/29 · 22:04 | Card disputes · golden set v7 | 411/420 | Pass | 18m 42s | |
| EV-2288 | 08/28 · 22:04 | Card disputes · golden set v7 | 414/420 | Pass | 18m 09s | |
| EV-2285 | 08/27 · 14:31 | Failure modes · adversarial v3 | 88/96 | Fail | 07m 55s | |
| EV-2281 | 08/26 · 22:04 | Account access · golden set v4 | 258/260 | Pass | 11m 20s | |
| EV-2276 | 08/25 · 09:12 | Confidence calibration · sweep | 1786/1840 | Pass | 42m 18s | |
| EV-2270 | 08/24 · 16:47 | Policy grounding · Reg E regression | 141/148 | Fail | 09m 33s |
Adversarial failure-mode suite
Three scripted degradations verify retry, fallback, confidence-floor, and consensus guardrails end to end.
3 scenariosOpen failure modes