This is the synthetic-query idea from the field guide's §05, made concrete: for every doc in the
corpus, ask an LLM to invent a few realistic questions that doc answers, batched the same way
cmd/buildindex already batches the corpus for catalog summaries.
1const genInstruction = `You are building training data for a document retrieval model. You will be given a batch of documents, each as a path and its full (or excerpted) content. 2 3For EACH document, in the same order as given, invent %d realistic questions a real user might type into a search box, such that THIS document is the correct answer. Vary the phrasing and specificity across the %d questions for a given doc (e.g. one terse keyword-style query, one full-sentence question, one from a beginner's phrasing) — don't just reword the doc's own heading. 4 5Respond with ONLY a JSON object of the shape {"entries": [...]}...`
The "don't just reword the doc's own heading" line is doing real work: without it, an LLM's path of least resistance is to reword the doc's own heading — training the model to match surface strings instead of meaning, which is exactly the grep-shaped behavior this whole project is trying to get past.
One query becomes one training example
1for _, e := range got { 2 for _, q := range e.Queries { 3 q = strings.TrimSpace(q) 4 if q != "" { 5 pairs = append(pairs, Pair{Query: q, Doc: e.Path}) 6 } 7 } 8}
A doc with 3 generated queries becomes 3 separate (query, doc) rows — flattened,
because the training loss (next section) operates on one query/one positive-doc at a time, not
on a doc with a list of valid queries attached.
1// dropLeakedPairs removes any training pair whose query is a 2// near-duplicate (token Jaccard similarity > 0.7) of a real eval query, … 3func dropLeakedPairs(pairs []Pair, evalQueries []string) []Pair { 4 … // tokenize every eval query once into evalTokenSets 5 for _, p := range pairs { 6 pTokens := tokenize(p.Query) 7 leaked := false 8 for _, evalTokens := range evalTokenSets { 9 if jaccard(pTokens, evalTokens) > 0.7 { 10 leaked = true 11 break 12 } 13 } 14 if !leaked { 15 out = append(out, p) 16 } 17 } 18 return out 19}
This is the two-tower diagram and the InfoNCE loss from the field guide's §02 and §05, as an
actual training loop. There is surprisingly little code — nearly all of it is
sentence-transformers doing the batching and gradient step; the file mostly just
wires your pairs and your GPU into it.
Every pair becomes one training example
1examples = [ 2 InputExample(texts=[args.query_prefix + p["query"], doc_text(p["doc"])]) 3 for p in pairs 4]
InputExample(texts=[a, b]) is sentence-transformers' way of saying "these
two texts are the anchor and the positive" — literally the query dot and the green document
dot from the embedding-space diagram, paired up.
1train_loader = DataLoader(examples, shuffle=True, batch_size=args.batch_size) 2loss = losses.MultipleNegativesRankingLoss(model)
This one line is the InfoNCE formula from the field guide:
MultipleNegativesRankingLoss computes this, with one detail the formula hides: the
cosine sim is multiplied by a scale (default 20) before the exponent,
otherwise a similarity in [−1, 1] would give a nearly flat softmax. It gets its
negatives — the "Σᵢ" over all dᵢ — for free from whatever else happens to be in the
batch. With batch_size=32, every query trains against 31 negatives it never had to
be told about explicitly; double the batch size and every example gets more of them at no extra
data-collection cost (though, as The Best Wrong Answer works
through, only the negatives that are actually confusable contribute much gradient). This is also the number that trades most
directly against your 10GB VRAM budget — bigger batches need more memory.
Running it on the GPU
1model = SentenceTransformer(args.base, device="cuda") 2… 3model.fit( 4 train_objectives=[(train_loader, loss)], 5 epochs=args.epochs, 6 warmup_steps=warmup_steps, 7 show_progress_bar=True, 8 output_path=args.out, 9)
This is literally the "step 1 / step 2 / step 3" loop from the training animation —
.fit() just runs it over and over — 1,372 pairs at batch 32 is 43 steps per epoch, 172 over the default 4 epochs: sample a batch, compute the loss above,
take a gradient step that pulls each query closer to its true doc and (implicitly, via the
softmax denominator) away from the other 31 docs in the batch. output_path is where
the nudged weights land — this is the file eval_embed.py loads next.
coverage: none queries always count as misses because a bi-encoder can't abstain)This is the "compare vectors with a dot product" half of the two-tower diagram, done for all 45 eval queries against all 457 docs at once — the brute-force approach from §07, since 457 × 384 floats has no business needing a vector database.
1doc_vecs = model.encode(doc_texts, batch_size=32, show_progress_bar=True, normalize_embeddings=True) 2query_vecs = model.encode(query_texts, batch_size=32, normalize_embeddings=True)
Same split as the bi-encoder diagram: 457 docs through the encoder once (this is the offline pass — in a real deployment this line runs once and gets cached, not on every query), 45 queries through the same encoder, shared weights, separately.
1sims = query_vecs @ doc_vecs.T # (num_queries, num_docs) cosine similarity 2… 3order = np.argsort(-sims[i]) 4ranked = [doc_paths[j] for j in order[:10]]
Here that shape is (45, 457). Because both sides were normalize_embeddings=True, a plain dot product
is cosine similarity — the @ is the "·" node from the bi-encoder diagram,
just batched into one (45 × 384) @ (384 × 457) multiply instead of 45×457 separate
comparisons. This is also the line that would need to change to FAISS/HNSW if the corpus
ever grew past the point where numpy can do this in microseconds.
The same hit/acceptable rule as the Go evalrun harness
1hit = paths_equal(top1, ideal) 2acc = hit or any(paths_equal(top1, a) for a in acceptable_docs)
Deliberately copied from score() in agent/cmd/evalrun/main.go,
not reinvented — hit requires the exact ideal_doc,
acceptable also allows anything in acceptable_docs. Same rule, same
eval set, means a bi-encoder's score lands on the same scoreboard as the graph agent and
single-shot (+catalog) agent methods already scored the same way: 23/45 and 26/45 exact hits.
Check your understanding
1. What does the leakage guard in gentrainpairs actually check for?
dropLeakedPairs compares each training query's tokens against every eval query's tokens; a Jaccard similarity above 0.7 means the two are close enough to count as leakage, and the pair gets dropped.
2. Why does increasing batch_size make MultipleNegativesRankingLoss's training signal "harder" per example?
batch_size=32 gives every query 31 negatives for free, just by sharing a batch with other examples — no extra mining or labeling required. Double the batch, double the free negatives.
3. Why can eval_embed.py score all 45 queries against all 457 docs with one matrix multiply instead of a vector database?
A (45 × 384) @ (384 × 457) matrix multiply is nothing for a CPU, let alone a GPU. Reaching for FAISS/HNSW only pays off once the corpus (and thus the matrix) gets large enough that brute force stops being "instant."
4. What's the difference between a "hit" and an "acceptable" score in this repo's evaluation rule?
Both are copied straight from agent/cmd/evalrun/main.go's score(): hit is strict (top1 == ideal_doc), acceptable is hit OR top1 is anywhere in acceptable_docs — the same rule every method in this repo is scored by.
5. Why is normalize_embeddings=True required for query_vecs @ doc_vecs.T to equal cosine similarity?
Dot product = ‖q‖‖d‖cos θ in general. Cosine similarity is just cos θ — you only get that directly from the dot product once both vectors have length 1, which is exactly what normalize_embeddings=True guarantees.