Performance and cost#

Venn is designed to make repeated decisions over existing embeddings low in latency and incremental compute cost. Its decision layer reuses a representation that may already have been computed for search, routing or another task.

Model footprint#

Venn's generic decision head is 787,457 parameters, about 3.15 MB in float32. It runs on CPU. With existing vectors, the reader can stay unloaded; the outcome-store path also runs without the generic head.

For fresh text, Venn disables EmbeddingGemma 2's vision and audio towers with config_kwargs={"vision_config": None, "audio_config": None}. Google's model card gives the text-only model as approximately 270 million parameters. Google's launch announcement reports that quantized text-only weights can occupy about 191 MB of active RAM on a Pixel 11 Pro; the full multimodal model occupies about 567 MB in that deployment.

Venn's current reader path uses PyTorch, with float32 on CPU. Google's smaller quantized deployment uses a different execution path; Venn does not currently provide that backend. Weight storage uses decimal MB/GB and excludes runtime working memory. Google's active-RAM numbers also refer specifically to weights, not an entire application's memory use.

The pinned checkpoint at revision f9972ac22e696a105c482202646979e33e4b5a1e has a 1,488,915,288-byte safetensors file. Its text/shared tensors account for 542,005,296 bytes at BF16. To inspect the cached file without loading its weights into a model, substitute the path printed by venn download-embedder:

from math import prod
from pathlib import Path
from safetensors import safe_open

path = Path("/path/to/reader/model.safetensors")
elements = 0
with safe_open(path, framework="numpy") as weights:
    for name in weights.keys():
        if "audio" not in name and "vision" not in name:
            tensor = weights.get_slice(name)
            assert tensor.get_dtype() == "BF16"
            elements += prod(tensor.get_shape())
print({"text_shared_elements": elements, "bf16_bytes": elements * 2,
       "checkpoint_bytes": path.stat().st_size})

Alongside other local decision models#

The footprint advantage also matters against open-source language-model-based alternatives. Kev's model list starts at 0.8B parameters and includes 4B, 9B and 27B versions; Open JEV uses Qwen3-4B. Venn's approximately 270M-parameter text reader is about three times smaller than a 0.8B model and 15 times smaller than a 4B model by parameter count. Quantized files and total runtime memory depend on each implementation.

The architectural saving is the small decision computation after embedding. Fitted linear heads score stored vectors directly; outcome decisions query the store. The arena head answered an embedded screen in under 0.5 ms in the recorded run. The wine price head fit and scored 65,701 stored vectors in about 7 seconds.

Local alternatives can also cache content, and projects such as jevos explicitly target CPU deployment. Venn has not been timed against these projects on a common workload. Compare their complete input-to-answer paths, including embedding and task accuracy, before assigning a relative speedup.

The reader can be replaced independently of the outcome decision API. A different reader can trade size, speed and quality without requiring a different application-level decision loop.

Where the savings can come from#

Consider a ticket that needs a team, a priority and a topic. Its content can be encoded once, then reused by each question. For a collection whose documents are already embedded, a new question requires preparing its instructions/options and scoring the stored vectors. It does not require re-encoding every document. All vectors must belong to the supported profile.

The generic head has 787,457 trainable parameters: about 3.15 MB of FP32 parameter storage. This excludes model buffers, the embedding reader, loaded libraries, vector stores and request working memory. Vector-only serving can execute on CPU without loading the reader. Raw text, including new textual criteria, still requires encoding.

A workload cost model#

For N items and Q questions applied to each item, a simplified cost decomposition is:

Total work = content encoding
           + question preparation
           + N × Q × per-decision scoring
           + vector I/O and result delivery
           + any family fitting

For an existing compatible vector collection, content encoding has already been paid. For fresh content, amortize it over the actual number of decisions. Report both total and incremental costs. Convert measured compute time to dollars using the actual hardware/provider rate, including utilization and idle capacity; local inference still consumes resources. The API reports zero in its token and cost usage fields; they are not instrumented.

This is a cost-accounting model. Jev, TypeSafe's hosted decision model whose Decisions API request format Venn accepts, supports multiple typed questions in one request; comparisons should use that capability and any relevant caching. See TypeSafe's interface description.

Measure the CPU sweep#

After installing Venn, run:

venn sweep --rows 1000 --batch-size 256
venn sweep --rows 1000000 --batch-size 8192

The larger run creates a temporary 1.536 GB float16 vector file plus working memory. It uses seeded synthetic vectors and headless centered-cosine scoring. No reader or head download is needed. Every output is consumed and hashed.

Read generation_seconds, scoring_serialization_consumption_seconds, rows_per_second, end_to_end_seconds and counts. Counts must sum to rows. Throughput covers scoring, serialization and consumption; generation is reported separately. The report excludes model/service loading and final report writing. This standalone path does not measure HTTP requests, a database adapter, learned abstention, or semantic quality. To exercise the actual index streaming service, which applies one prepared question to every vector in a registered index and streams the results as NDJSON, use the trained-head path in the combined demo.

What a performance comparison must report#

• Quality: identical tasks and inputs, with accuracy and abstention coverage measured alongside speed. An inexpensive wrong decision is not a useful saving.
• Latency: cold startup, first request and warm p50/p95 latency, with batch size and concurrency. Aggregate rows/s is not single-request latency.
• Throughput: rows and questions per second, including full output consumption.
• Resources: CPU/GPU model, thread count, memory, software versions and power or hosting cost where measured.
• Accounting: reader, preparation, fitting, vector I/O, serialization and transport costs; distinguish already embedded inputs from fresh inputs.

The evidence ledger records current comparisons. A wine head was built in 31 seconds on one A100; a separate five-round Jev discovery run took 17.4 minutes and produced 35 questions. The accuracy comparison uses the 38-question pipeline. For price, fitting and applying a head over 65,701 already embedded notes took about 7 seconds. See Venn and Jev for the workflows and results.

Reuse follows the representation. The winning wine score head uses SentenceSimilarity embeddings and text length inputs; the price comparison uses Classification embeddings. A family requiring text inputs cannot consume a vector-only index. Kernel ridge compares query rows with its fitting vectors; its cost grows with the fitting set. Linear ridge and Choice heads are fixed projections suitable for matrix scoring across a collection.

Development measurements of the in-process outcome API are recorded in the development evidence.

View this page on GitHub