VennComparative evaluation protocol#
Status: proposed experiments, not measured results or a preregistration of completed work. Freeze a versioned protocol and all split identities before running the comparisons below. VennSpace-1 remains unchanged.
Question#
What does a learned Venn head or registered family add to decisions made from the same frozen embeddings, and when does reuse reduce total workload cost? Keep the contribution of the reader distinct from that of the decision layer.
Matched comparisons#
| Workload | Required controls | Measurements |
|---|---|---|
| Routing with no-match examples | Raw/centered cosine, nearest reference, centroid, linear classifier, learned abstention and registered family | Accuracy, macro-F1, risk versus coverage, no-match precision/recall, Brier score and calibration intervals |
| Ordered scoring | Cosine ordering, linear/ridge ordinal baseline, registered family | Spearman rho, absolute error, ordering violations and calibration where meaningful |
| Repeated decisions over cached rows | Same batches/options on headless and trained-head paths | Cold/warm latency, p50/p95, memory, rows/s, serialization and full consumption |
| End-to-end decision systems | Relevant semantic router/few-shot classifier and optional generative decision baseline | Quality and abstention at comparable coverage, encoder cost, fitting cost, repeated-query amortization and cost per accepted correct decision |
For Jev, evaluate multiple questions in one request where supported. For every baseline, use its relevant caching/batching capabilities. Report fresh-content and already-embedded conditions separately, with cold/warm latency, throughput, peak memory and actual compute/provider cost. Include vector-store I/O and serialization. Follow the cost-accounting boundaries.
Use identical official test rows and training budgets where methods permit. For methods that tune the encoder, report that as a separate condition rather than attributing the gain to a head. Specify versions, prompts, label budgets, seeds and parameter choices before evaluation. Compare within method scope; MTEB and ColBERT are useful evaluation/architecture references, not interchangeable routing baselines.
Splits and uncertainty#
Separate fitting, early stopping, probability/threshold selection and final reporting. Account for overlap with the generic head's training datasets; Banking77 and CLINC are not automatically unseen tasks. Include new domains and genuine no-match examples. Group duplicates and related variants before splitting. Choose the resampling unit at the independent example/task level, not at the paraphrase level. Report all chosen tasks and failed runs.
Ablate centered options, learned abstention, family residuals and calibration. For K=2 record the runtime's fallback decision. For ordered scoring separately validate label order; a nominal classifier is not an ordinal model by default.
Required release artifacts#
Each run must identify source commit, protocol hash, reader and head hashes, source revisions, split hashes, runtime versions, hardware, seeds, commands, metrics and uncertainty, resource consumption and exclusions. Make an aggregate reference and comparison command public. Document any unavailable training recipe or checkpoint explicitly. No success threshold should be chosen after examining final test outcomes.