Reproduction#

There are three distinct checks: executable contracts, headless scientific measurements, and trained-head results. Passing one does not establish the others. See the evidence ledger for the availability of each claim.

CPU contracts and packaging#

From the checkout, using Python 3.12:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
python -m pip check
python scripts/check_docs.py
venn quickstart
venn sweep --rows 19 --batch-size 8
python -m pytest -q
python -m build

Tests use deterministic fake embeddings and untrained fixtures. They require no real reader or head download. Optional adapter tests skip without their extras. CI also exercises upstream clients and the installed wheel outside the checkout. Use getting started for CPU-only PyTorch setup.

Fresh headless scientific run#

Use an isolated virtual environment so pinned versions are not shadowed by user-site packages. CUDA is required by the current fresh-embedding worker. Choose a visible GPU count; the default is one. Preflight checks devices, required distributions and output/cache writability before dataset or reader downloads.

python3.12 -m venv /tmp/venn-science-env
/tmp/venn-science-env/bin/python -m pip install '.[embedding,benchmark,dev]'
/tmp/venn-science-env/bin/python -m pip check
/tmp/venn-science-env/bin/venn download-embedder
/tmp/venn-science-env/bin/venn benchmark \
  --output /tmp/venn-space-run --cache /tmp/venn-fresh-cache.sqlite --gpus 1
/tmp/venn-science-env/bin/python scripts/compare_reference.py /tmp/venn-space-run/summary.json
/tmp/venn-science-env/bin/venn redteam \
  --output /tmp/venn-boundary-run --cache /tmp/venn-fresh-cache.sqlite --gpus 1
/tmp/venn-science-env/bin/python -m pip freeze > /tmp/venn-science-environment.txt

Run these commands from the root of the source checkout: venn benchmark reads its protocol from benchmarks/vennspace-v1.json, which the wheel does not contain, and the comparison script lives in scripts/.

Read the pinned reader's upstream terms before downloading. Original datasets are fetched at the revisions in the frozen protocol. Output/cache paths must be new for a fresh run. Sharing the newly generated cache with the boundary suite is explicit reuse within the same run; copying another person's embeddings is not independent fresh-embedding reproduction.

The reference run used four A100 GPUs. Minimum VRAM, RAM, disk, elapsed time and download size have not been qualified for other hardware. The code reduces embedding batches after CUDA out-of-memory errors, but this does not make every GPU suitable. Changing GPU count preserves task definitions.

Outputs and recovery#

File	Contents
`environment.json`	Python/platform, key package versions, CUDA/device names and invoked command
`protocol.json`	Frozen task or authored-suite definition
`*-splits.json`	Original source rows, labels and normalized-text hashes; no corpus text
`*-predictions.npz`	Benchmark predictions, targets and sampled row identities
`summary.json`	Geometric metrics, intervals, protocol hash and embedding telemetry
`results.json`	Boundary-suite or combined-demo outputs; authored example text may be present

Completed embedding batches are committed to the SQLite cache. Repeating a command with the same paths reuses exact cache keys; a changed frozen protocol is rejected. Final result files are overwritten on a rerun, so use a new output directory to preserve comparisons. environment.json describes the latest invocation. Archive the protocol, outputs, environment and file checksums after a successful run. Never describe a cache replay as fresh embedding.

compare_reference.py reports each metric's observed value, reference value, difference, interval and pass/fail. It exits 0 if all ten are within the rounded reference intervals, 1 for a mismatch, or 2 for invalid input. This reproduces the documented consistency criterion, not a claim of statistical equivalence. The reference JSON is transcribed from the existing table, not raw-run precision.

Trained-head evaluation#

The venn-1 head is published on Hugging Face at markrussinovich/venn-1 (revision e618a6919e5a15c6ced46c9874424363c2955283); venn download-model fetches it and prints the assets path. The family example exercises the downloaded head. There is no public ordinal evaluation runner yet. Follow the comparison protocol before making new model-quality claims.

Upstream client validation#

First terminal, from the checkout with the dev environment active:

python scripts/contract_server.py --port 8000

Second terminal, with that environment active (Node 24 is used by CI):

python examples/route_ticket.py
python examples/index_vectors.py
cd tests/clients
npm ci --ignore-scripts
npm run contract -- http://127.0.0.1:8000
node playground.mjs http://127.0.0.1:8000
python -m pip install -r requirements.txt
python plugin.py http://127.0.0.1:8000

The contract server uses fake embeddings and temporary untrained head fixtures. These outputs validate interfaces, not semantic routing. The upstream-client checks restrict SDK traffic to the local origin. Stop the server with Ctrl-C.

View this page on GitHub