Compare · bake-off
bezoar vs Company B
A bootstrapped open-source CLI. Company B publishes no detection benchmark, so we ran it on the same 1,431-sample public corpus as bezoar, default config: 25.4% of 626 known-malicious skills flagged (bezoar 88.2%) and 391 false positives on 737 official vendor skills (bezoar 0). On the small curated fixture: bezoar 13/13, Company B 6/13; benign set 0/6 vs 2/6 flagged.
Measured · same corpus · default configs
| Metric | bezoar | Company B |
|---|---|---|
| Detection rate (626 malicious, public corpus) | 88.2% | 25.4% |
| Flagged, absolute | 552/626 | 159/626 |
| False positives (737 official vendor skills) | 0 | 391 |
| False positives (180 MCP Registry configs) | 0 | 0 |
| Curated fixture (small set) | 13/13 | 6/13 |
| Curated benign set flagged | 0/6 | 2/6 |
What the number means
- Both of Company B's numbers are dominated by one Critical substring rule that fires on
eval,exec, orshellanywhere in the text.evalis a substring ofevaluation; “when shell access is available” in a vendor README is a Critical. That one rule accounts for most of the 391 official-skill false positives and most of the 159 malicious hits. - MCP and prompt-injection coverage is thin: 18 of 293 shipped rules are prompt rules, and they look for “ignore previous instructions” shapes. Mandatory pre-step tool poisoning and conceal-from-user shadowing are not in its taxonomy.
- Method: measured by us, default config, all phases, no login or enhanced mode, one isolated scan per sample, no tuning for either tool. “Flagged” is a High or Critical verdict, which is Company B's own failing threshold. Corpus and results JSON are public.
Being written
This page is in progress. The source of truth is the repository until it lands. Back to home.