VennResults and evidence#
This ledger lists each headline claim, the method and artifacts behind it, and how to reproduce it. It is not a new evaluation.
| Claim | Method / artifact | Reproduction | Status |
|---|---|---|---|
| Ten VennSpace-1 geometric metrics | Protocol, rounded reference JSON, methodology | venn benchmark --output /tmp/venn-space --cache /tmp/venn-space.sqlite --gpus 1 then python scripts/compare_reference.py /tmp/venn-space/summary.json |
Public headless code; original reader/data fetched separately |
| Sixteen semantic boundary rates | 32 authored cases in venn/redteam.py; reference table |
venn redteam --output /tmp/venn-boundary --cache /tmp/venn-boundary.sqlite --gpus 1 |
Public headless code; finite authored suite |
| All 26 rates within reference CIs in an independent run | Summarized in benchmark and boundary findings | Requires a new independent fresh-embedding run | Reported result; the original environment and raw-run archive are not distributed here |
| 787K generic-head parameters / 3.15 MB FP32 | Head architecture, model card | Run venn download-model, then load the head and count tensors |
Architectural size, excluding reader/runtime; not a speed or quality result |
| Text reader: approximately 270M parameters, about 540 MB at 16-bit | Google model card, pinned checkpoint tensor headers | Footprint inspection | Text/shared tensors: 542,005,296 BF16 bytes; default CPU loads float32 |
| Quantized reader: about 191 MB active RAM for text-only weights | Google launch announcement | Deployment configurations | Google measurement on Pixel 11 Pro; separate from Venn's current PyTorch reader |
Development measurements of the outcome API are recorded in the development evidence.
Comparative results#
The Venn and Jev guide explains when these results matter. The table below is the index to the current comparisons and runnable examples.
| Task / metric | Venn | Jev / comparison method | Evaluation scope | Source and how to try it |
|---|---|---|---|---|
| Support routing, Banking77 · accuracy ↑ | References: 93.6% | Jev 1.13: 79.7% | 3,080 tickets | Precedent Desk, its eval.py runner and recorded manifest |
| Support routing, CLINC150 · accuracy ↑ | References: 94.3% | Jev 1.13: 91.1% | 4,498 tickets | Second-dataset results and runner |
| CFPB refunds · AUROC ↑ | Neighbors: 0.768 | Jev-label historical rates: 0.702; trained Jev features: 0.816 | Predict outcomes using complaint history; the trained Jev-feature model scores highest | History walkthrough, numeric record; outcome-store example |
| SEC industry groups · accuracy ↑ | Linear family: 73.3% | Jev: 64.1% | 1,000 additional reports; report-only confirmation. The original 60-filing accuracy and useful-answer gates failed | SIC walkthrough, numeric record; linear-family recipe |
| Wine critic score · RMSE ↓ | Kernel family: 1.803 | Jev 38-question pipeline: 1.794 | 3,000 fresh notes; cookbook runner uses a separate 1,200/800 split | Wine walkthrough, numeric record; python examples/wine/wine.py --form kernel_ridge |
| Wine price · log-price RMSE ↓ | 0.538 | Direct Jev: 0.531 | Accuracy and cached-vector runtime are separate measurements | Price record and workflow |
| Audio emotion, CREMA-D · accuracy ↑ | Audio linear family: 56.5% | Jev on transcripts: 14.6% | 1,716 clips; inputs differ: sound versus words | Sound walkthrough, numeric record; audio example |
| Grape naming · macro-F1 ↑ | 0.340 | Jev: 0.427 | November 6 comparison write-up; full evaluation runner not included | Comparison discussion |
| Firefox component routing · accuracy ↑ | Head: 0.583 | Jev: 0.641 | November 6 comparison write-up; full evaluation runner not included | Comparison discussion |
↑ Higher is better; ↓ lower is better. These are separate task evaluations, not a single aggregate benchmark.
Runtime and arena results#
| Measurement | Recorded result | Scope / caveat | Source |
|---|---|---|---|
| Wine predictor build | Venn: 31 s on one A100; Jev discovery: 17.4 min | Separate five-round discovery run produced 35 questions; accuracy above uses the 38-question pipeline | Build record and timing boundaries |
| Wine price fit and prediction | About 7 s for 65,701 cached notes | Venn fitting and scoring on already embedded inputs; excludes fresh embedding | Price workflow and record |
| Arena action selection | Compiled head: 8.36 kills/game; memory: 6.20 | 250 scored games; comparison is head versus memory, not Jev | Arena results and runner, recorded manifest |
The comparison notebooks walk through recorded results. The wine, support desk, audio and arena examples run the respective Venn workflows on source or local data. Detailed experiment records stay on the methods page. Scientific benchmark commands above continue to use the frozen VennSpace-1 protocol.
Interpreting reproduction#
| Reproduction detail | Meaning |
|---|---|
| Reference JSON | Transcription of the published two-decimal percentage table, with the frozen protocol hash; original raw measurements were not recovered |
compare_reference.py |
Checks each new point estimate against the rounded reference intervals |
Exit 0 |
All ten metrics are inside the intervals |
Exit 1 |
At least one metric is outside the intervals |
Exit 2 |
Input is invalid or incompatible |
| Interpretation limit | Consistency with rounded intervals does not establish statistical equivalence, generalization, or a learned-head advantage |
New runs emit predictions and source-row manifests locally. Source corpora, reader weights and caches are not part of this repository. Record software versions, hardware and fresh/cache status when reporting results. Do not relabel synthetic sweeps as semantic evaluation.
Further comparisons are specified in the comparison protocol. The performance and cost guide explains which work is reused, which expenses remain, and how to measure latency and throughput.