VennSemantic boundary findings#
Embedding similarity is not reasoning. Venn is therefore not a guardrail, authorization mechanism or numeric rule engine, and this suite shows where it fails. API compatibility and probability normalization do not repair these failures.
The frozen public suite (venn redteam --output /tmp/venn-redteam
--cache /tmp/venn-cache.sqlite --gpus 1) contains 32 authored cases across
support routing, sentiment, numeric comparison and explicit refund negation.
It evaluates 288 intent-preserving/paraphrase/visual-format candidates and
64 separate instruction-conflict candidates. All candidates are evaluated:
the finite boundary search chooses the worst outcome per original case.
Paraphrases are authored and labeled, not independently human-certified.
This is a limited fixed suite, not a robustness certification.
| Task | Cases | Baseline correct | Any intent-preserving candidate flips | Worst-case correct | Instruction-conflict flip |
|---|---|---|---|---|---|
| Support routing | 12 | 100% | 8.33% | 91.67% | 8.33% |
| Sentiment | 12 | 83.33% | 16.67% | 83.33% | 75% |
| Numeric comparison | 4 | 50% | 50% | 25% | 50% |
| Explicit negation | 4 | 100% | 25% | 75% | 25% |
These are local reference measurements for the venn-1 profile, not estimates
of broad population accuracy. The original case, not each correlated candidate,
is the interval unit for Wilson 95% intervals. Small four-case intervals are
especially wide. The runnable suite writes detailed outputs to the directory
you supply, outside the checkout. A flip can correct an initially wrong answer;
it is not synonymous with a new error. Instruction-conflict variants are not
counted as meaning-preserving.
Scoring uses public centered-cosine conditional Choice ordering. Retain an explicit abstention/fallback path and validate on your own labeled task and threat model. Do not treat arbitrary instructions as a general reasoning capability, untrusted suffixes as harmless, or similar embeddings as proof of identical meaning. Unknown examples can be worse than this suite.
An independent rerun of the same suite in a fresh isolated environment on four A100 GPUs with fresh embeddings matched all sixteen aggregate rates within their reference intervals.