English · 한국어

Verification¶

This page records how vecscope's claims were checked and what the measurements show, in five areas:

1. Algorithm exactness: the k-NN, the incremental audit and the published tables against float64 and against a full recomputation
2. Thresholds and retrieval quality: every finding against nDCG@10 on labeled data, the query findings, and what retrieval does when a plan step is applied
3. Sampled audits: what a sample can and cannot stand for
4. ANN recall and index sweeps: the recall scorer and the sweep
5. MCP security and operations: the server's guards and agents following the plan over MCP

The thresholds and the plan rules are set from these results. Each area ends with what it does not cover; the limitations collect them.

• Environment: AMD EPYC 7R13, 8 vCPU, 26 GB RAM, NVIDIA L4; Python 3.12.3, numpy 2.2.6, faiss 1.15.1, sentence-transformers 3.4.1, torch 2.6.0
• Corpora:
• six synthetic shapes (examples/bench_common.py: clustered, isotropic, anisotropic, duplicates, lowrank, zipf)
• the eight public pre-computed embedding sets of the README table
• nine labeled datasets encoded with 19 models: BEIR SciFact, NFCorpus, ArguAna, SCIDOCS and FiQA-2018, and four Korean sets (ko-strategyqa, and from MTEB XPQA kor-kor, AutoRAG and MultiLongDoc ko)
• Vector stores: faiss 1.15.1, Qdrant client 1.20 (local mode), Chroma 1.5, Milvus Lite 3.2, Weaviate client 4.24
• Tests: pytest -q and pytest -m slow pass; the vector store tests run against real local Qdrant, Chroma and Milvus Lite instances and a faiss index file

1. Algorithm exactness¶

Not covered: exactness on real corpora over 20,000 rows was checked on synthetic data; the real-data check covers the 76 smaller pairs.

2. Thresholds and retrieval quality¶

Finding values against nDCG@10¶

Spearman correlation between each finding's value and nDCG@10 across the 19 models, per dataset. A negative value means the finding rises as retrieval gets worse.

ArguAna nDCG@10 is computed as BEIR does, without the query's own document, which is in the corpus.

What this supports:

• hubness and query_hubness are negative on all six datasets (−0.03 to −0.90). They carry a signal about retrieval quality across models, strongest where some models do not handle the language or domain at all (ko-strategyqa, ArguAna, SCIDOCS).
• query_gap is negative on all six (−0.02 to −0.81) but weak on English data, where every reasonable model keeps it between 0.9 and 2.6 (ArguAna 0.05 to 0.93, because its queries are arguments taken from the corpus). Its job is to catch queries from another space, not to rank models.
• anisotropy and rogue_dimensions change sign between datasets; they are not quality scores. anisotropy is ok on all 114 pairs.
• ArguAna, SCIDOCS and FiQA were not used to set any threshold. On them:
• query_gap flags only GloVe on SCIDOCS (2.90, warn); good models stay at or below 1.59.
• query_hubness warns on 17 of the 19 models on FiQA, the good ones at 4.0 to 4.2, just above the warn threshold of 4 because the skewness grows with corpus size (English models: 1.5 to 3.4 up to 26k documents, 4.0 to 4.8 on FiQA's 58k).
  • The thresholds were kept, since refitting them on a held-out set would make it a training set; the explanation states the range instead.
  • It also fails the multilingual models on SCIDOCS (14 to 25, nDCG@10 0.15 to 0.17 against 0.23 for the best English models).
• hubness warns on most models on SCIDOCS (25k documents) and FiQA (57k): skewness grows with corpus size, so compare it against a baseline of the same corpus rather than read the absolute warn.
• A pipeline that is wrong in an obvious way (GloVe on BEIR, a Korean-only model on English, an English-only model on Korean) raises several findings at once. The count of warnings separates such pipelines; a single fail does not.

Korean data and document length¶

Three more Korean sets from MTEB, none used to set a threshold, differ mainly in document length. Documents were truncated to 512 tokens for every model.

The good models are KURE-v1, BGE-m3-ko, bge-m3, multilingual-e5-base and -large, gte-multilingual-base and Qwen3-Embedding-0.6B.

• Long documents (MultiLongDoc). The findings behave as on BEIR.
• hubness or query_hubness flags all 9 English-only models.
• The good models are clean except one hubness and two query_hubness warnings (and the deprecated ood_queries, which still flags the e5 models).
• Spearman with nDCG@10: query_hubness −0.86, hubness −0.76, query_gap −0.80.
• A narrow-domain corpus (AutoRAG, finance and commerce documents). The ranking signal holds, but one default does not fit.
• query_hubness correlates at −0.94 and, with hubness, flags 6 of the 9 English-only models; no good model is flagged by either.
• effective_rank warns on all 7 good models (0.064 to 0.094 against a warn of 0.10): a corpus about a few topics uses few directions. This is the domain effect vecscope calibrate exists for, not a model fault.
• Very short documents (XPQA product answers). The corpus findings describe the data, not the pipeline.
• Every model, good or not, fails collapse (0.15 to 0.47) and effective_rank, and 15 of 16 flag query_gap. Only 4% of the answers have identical text, but answers such as "네, 가능합니다" and "네, 가능해요" embed almost identically in every model.
• hubness warns on 6 of the 7 good models and on none of the English-only ones, so its correlation with nDCG@10 turns positive (+0.43).
• Read this as "many documents are interchangeable for retrieval", which the low nDCG@10 of every model confirms. The plan puts query_gap with the duplicates, so the queries are not blamed for it.

Query findings¶

query_gap and query_shift were chosen from three candidates measured on the cached embeddings of the 19 models on SciFact, NFCorpus and ko-strategyqa. Every (model, dataset) pair was scored on its own queries and on three injected faults:

• the query prefix dropped (bge, e5, mxbai; 21 sets)
• queries embedded by every other model of the same dimension (312 sets)
• English-only models on Korean data (11 sets)

q_nn is the nearest query-to-corpus distance. The thresholds sit just above the worst reproduced pipeline, so they are fitted to this data without a held-out split.

query_gap needs no reference; query_shift compares against the query distances stored in a baseline. ood_queries stays in the output for comparison and is deprecated.

Three of the fault criteria turned out to describe something other than a query-side fault:

• Missing prefix. Dropping the bge, e5 or mxbai query prefix changed nDCG@10 by −0.014 to +0.007 on SciFact and NFCorpus, and moved the query distances closer (bge-base query_gap 1.55 to 1.21 on SciFact) or farther (e5-base 1.64 to 1.75 on NFCorpus).

There is no geometric signal to detect, and on this data little harm. - Another model. Models fine-tuned from the same base share a space: the mean cosine of the same document under mxbai-embed-large and bge-large is 0.985, under gte-base and bge-base 0.885 (0.004 for mpnet and bge-base).

Their swapped queries keep most of the nDCG@10 (gte-base queries on the bge-base corpus: 0.697 against 0.740), and they are the two 768-dim swaps that query_gap and query_shift did not fail. - Another language. An English-only model on ko-strategyqa encodes the Korean queries and documents equally badly, so nothing is asymmetric between them; query_hubness fails 10 of the 11 (11 to 30) and warns on the last (8.3).

Interventions¶

Does retrieval quality move when a plan step is applied?

The verification harness (kept outside this repository, like the other measurements on this page):

• takes 21 real (model, dataset) pairs (8 models on SciFact, 8 on NFCorpus, 5 on ko-strategyqa)
• applies each change to the corpus and the same transform to the queries
• records the findings, the plan and retrieval (nDCG@10, recall@10, duplicate share) before and after

What this supports:

• A finding the plan acts on and a retrieval change move together when the defect is real: deduplicating, z-scoring a dominant coordinate and raising ef_search each recover retrieval quality, and ann_recall ranks index settings the way ranked nDCG@10 does.
• Hub reduction helps where the finding says there are hubs: CSLS against the queries raised nDCG@10 on every pair with query_hubness not ok and was a coin flip elsewhere, so the plan proposes it only for non-ok hub findings and the re-measure line asks for labeled queries.
• Search latency: on these corpora (5,000 documents or fewer) one batch of queries takes 0.002 to 0.04 ms per query at every ef_search, below the measurement noise, so the latency side of the ef_search trade-off is measured by vecscope sweep on 100k rows (README "Index sweep"), not here.
• Centering a healthy model does not help: it removed anisotropy and cost up to 0.034 nDCG@10. The plan only proposes it when anisotropy is not ok, and its re-measure line says to check retrieval on labeled queries. A plan step is a hypothesis to test on the corpus at hand, not a diagnosis; an agent that applies one must re-audit and compare retrieval before adopting it.
• A dominant coordinate also spreads the norms and splits cosine from inner product. Normalizing does not remove the coordinate, so the plan puts those findings in the dominant_dimension step with the z-score change.

Not covered: the thresholds and the query findings were fitted on the data above, with no held-out split, and false positives and misses were not tracked per model, language, document length or corpus size beyond the nine datasets here. On very short templated documents and on narrow-domain corpora some defaults (collapse, effective_rank, query_gap) flag good models; calibrate on such data before gating. Production query logs and private corpora were not measured.

3. Sampled audits¶

Not covered: sampled audits use uniform random rows. Stratified (per-cluster) sampling was considered and not adopted: an island must be smaller than max(k + 1, 0.5% of the rows) and isolated, so a sample misses small islands whatever the strata, while stratifying would bias the distribution findings. Exact copies are counted over every row instead.

4. ANN recall and index sweeps¶

Not covered: search latency on corpora under 5,000 rows is below the measurement noise; the latency side of the trade-off is measured by vecscope sweep on 100k rows (README "Index sweep"). Index types other than HNSW and IVF were not swept.

5. MCP security and operations¶

The server's guards are pinned by tests/test_mcp_security.py on small matrices:

• a corpus, queries or comparison file outside the allowed directories is refused, and a directory or a missing file is reported as a user error
• --max-read-mb refuses a file before it is parsed, 0 lifts the limit, and a value that is not a number is reported; --max-rows and --max-dim refuse a matrix (or a queries file) over the limit whatever its byte size, and the command-line options set them
• a network store is refused without --allow-network, and the error never carries the password, the API key or the other URI secrets; the timeout= option must be a finite positive number of seconds and at most 3,600
• a MemoryError while reading or auditing reaches the agent as a tool error that names the ways out (a sample, a smaller input, the limits) instead of ending the call silently
• a sampled response states evidence_scope, needs_full_audit and the scope in its summary line

Vector store reads were tested against real local Qdrant, Chroma and Milvus Lite instances and a faiss index file (tests/test_adapters.py), including the byte limit stopping a read.

Agent integration¶

• tests/test_agent_workflow.py drives the MCP server over stdio: a corpus with unnormalized vectors and a shared offset, audited with labeled queries.
• The loop applies the first plan step and re-audits until no injected defect is left.
• The plan asks for metric (normalize) and then shared_direction (center), after which anisotropy, metric_mismatch, collapse and norms are ok and nDCG@10 is not lower.
• The test also re-audits with baseline set to the previous audit_id and checks that baseline_diff lists the step's findings as improved, none of them as regressions, and no loss of nDCG@10.
• It found that a baseline made before a transform must not be the reference for query_shift (the centered queries all looked shifted).
• The reference is now used only for the same corpus, checked by corpus_fingerprint.
• The same task given to Claude Code 2.1 in headless mode (claude -p with the vecscope MCP server and --allow-dir), twice: 10 to 11 turns, 51 to 54 seconds.
• It followed the plan in order (metric, then shared_direction).
• It re-audited with baseline set to the previous audit_id without being told to.
• It took metric_mismatch, anisotropy, hubness, antihubs and collapse to ok with nDCG@10 unchanged at 1.0.
• It left the remaining outliers warn (0.011, threshold 0.01) alone: the rows were not broken vectors and one was a labeled answer.
• A case on real data (agent_case.md): SciFact with bge-small-en-v1.5, a batch ingested twice and a worker that skipped normalization, labeled queries and an inner-product production top-10. Two runs of Claude Code with the model claude-opus-5-5, 8 and 11 turns, 58 and 75 seconds.
• Told only the symptom, both runs found the 600 copies and the 2,581 unnormalized rows, checked them in the data, removed the copies while keeping the qrels positions, and normalized.
• Both re-audited with baseline and got ok on every finding. Production nDCG@10 went from 0.100 to 0.713, and exact search from 0.689 to 0.713.
• A second case on the same data with clean embeddings and an IVF index (nlist=128) searched with nprobe=1. Two runs, 7 turns each, 37 and 41 seconds.
• Both audited, found every finding ok and production nDCG@10 at 0.422 against 0.713 for exact search, and put the fault on the index.
• Both swept the index, re-measured nprobe at the production nlist, and chose nprobe=32 and 64. Production nDCG@10 went to 0.700 and 0.711, with no finding regressed.
• Both had to re-measure by hand because the sweep chose its own nlist (132), which led to sweep --nlist.

Not covered: the Weaviate adapter was tested only against an embedded server in opt-in tests (VECSCOPE_TEST_WEAVIATE=1); remote Qdrant, Milvus and Weaviate servers were not tested. The timeout= option and the scrubbing of client errors were tested against an unreachable Qdrant port and a stubbed client, not against a slow server. The row and column limits and the MemoryError path were tested on small matrices with the limits set low, not by exhausting a machine. The memory of a long-running server under many calls was not measured. The agent runs are one model on three tasks; they show the tools and the plan are usable, not that every agent follows them.

Limitations¶

• The thresholds and the query findings were fitted on the data above; there is no held-out split. Production query logs and private corpora were not measured.
• The real-model measurements cover nine datasets in two languages. On very short templated documents and on narrow-domain corpora some defaults (collapse, effective_rank, query_gap) flag good models; calibrate on such data before gating. Exactness on corpora over 20,000 rows was checked on synthetic data; the real-data exactness check covers the 76 smaller pairs.
• Sampled audits use uniform random rows (see sampled audits for why stratified sampling was not adopted).
• The Weaviate adapter was tested only against an embedded server in opt-in tests (VECSCOPE_TEST_WEAVIATE=1); remote Qdrant, Milvus and Weaviate servers were not tested. The timeout= option and the scrubbing of client errors were tested against an unreachable Qdrant port and a stubbed client, not against a slow server.
• The MCP guards were tested with low limits on small inputs, not by exhausting a machine, and a long-running server was not measured.
• The agent runs are one model on three tasks; they show the tools and the plan are usable, not that every agent follows them.

Reproducing¶

• pytest -q runs the verification tests (tests/test_new_features.py, tests/test_scale_exactness.py, tests/test_agent_workflow.py, tests/test_mcp_security.py); pytest -m slow adds the 100k x 384 exactness checks.
• python examples/benchmark_scale.py --rows 10000 50000 100000 --dim 768 --json scale.json and python examples/benchmark_sweep.py --rows 100000 --dim 768 --corpus clustered zipf --json sweep.json re-measure the tables and record the machine in the JSON.
• python examples/retrieval_quality.py runs the labeled-data study for SciFact and NFCorpus.
• The 19-model, nine-dataset, query-finding and intervention runs use a harness that is not shipped with the package. The behavior they set is pinned by tests instead:
• tests/test_query_findings.py (query_gap and query_shift)
• tests/test_plan_incremental.py (the plan rules, including the dominant-dimension and duplicate cases)
• tests/test_regression_injection.py (injected faults)