Benchmarks
Performance comparison of CAPA’s three tool exposure modes across 5 real-world scenarios, 10 trials each (150 runs total) on claude-opus-4-8.
Summary
Section titled “Summary”150/150
trials passed
$0.99
capa-cli total
$0.99
capa-ondemand total
$1.22
capa-expose-all total
Variants under test
capa-cli
toolExposure: none - agent reaches tools through capa sh. No MCP schemas in the prompt.
capa-ondemand
toolExposure: on-demand - two meta-tools (setup_tools, call_tool). Schemas loaded lazily.
capa-expose-all
toolExposure: expose-all - every tool's JSON schema published up-front. Vanilla MCP baseline.
Cost per run
Mean cost per scenario (10 trials each). Lower is better.
capa-cli capa-ondemand capa-expose-all
Scenarios
Slack Release Note slack - 13 tools / 47 KB
Discover the #releases channel, then post a brief friendly v1.2.0 note.
| Variant | Cost | ± SD | Total tokens | Quality |
|---|---|---|---|---|
| capa-cli | $0.1179 | 0.0121 | 78,991 | 87% |
| capa-ondemand | $0.1440 | 0.0011 | 90,236 | 90% |
| capa-expose-all | $0.1649 | 0.0325 | 88,168 | 83% |
Sentry Top Issues sentry - 23 tools / 81 KB
Build a top-5 errors digest for backend-api over 24h.
| Variant | Cost | ± SD | Total tokens | Quality |
|---|---|---|---|---|
| capa-cli | $0.2602 | 0.0201 | 152,437 | 100% |
| capa-ondemand | $0.2479 | 0.0184 | 131,326 | 100% |
| capa-expose-all | $0.3474 | 0.0569 | 143,553 | 100% |
GitHub Issue from Bug github - 108 tools / 161 KB
Open one GitHub issue describing a SQL-escape bug. Agent only needs create_issue out of 108 schemas.
| Variant | Cost | ± SD | Total tokens | Quality |
|---|---|---|---|---|
| capa-cli | $0.1465 | 0.0335 | 84,550 | 100% |
| capa-ondemand | $0.1719 | 0.0038 | 80,502 | 100% |
| capa-expose-all | $0.1596 | 0.0177 | 63,846 | 100% |
Incident Triage sentry+slack - 36 tools / ~128 KB
End-to-end on-call: query Sentry top issue, fetch event, hypothesize, post triage to #oncall.
| Variant | Cost | ± SD | Total tokens | Quality |
|---|---|---|---|---|
| capa-cli | $0.2783 | 0.0202 | 213,026 | 100% |
| capa-ondemand | $0.2251 | 0.0053 | 148,481 | 100% |
| capa-expose-all | $0.3755 | 0.0363 | 226,239 | 100% |
Refactor: Request ID no SaaS - control case
Add an X-Request-ID middleware + tests to a small FastAPI skeleton. Pure Edit/Write - no SaaS calls.
| Variant | Cost | ± SD | Total tokens | Quality |
|---|---|---|---|---|
| capa-cli | $0.1895 | 0.0132 | 128,056 | 99% |
| capa-ondemand | $0.1970 | 0.0128 | 138,438 | 99% |
| capa-expose-all | $0.1754 | 0.0173 | 119,932 | 99% |