This is the synthetic-query idea from the field guide's §05, made concrete: for every doc in the
corpus, ask an LLM to invent a few realistic questions that doc answers, batched the same way
cmd/buildindex already batches the corpus for catalog summaries.
1const genInstruction = `You are building training data for a document retrieval model. You will be given a batch of documents, each as a path and its full (or excerpted) content. 2 3For EACH document, in the same order as given, invent %d realistic questions a real user might type into a search box, such that THIS document is the correct answer. Vary the phrasing and specificity across the %d questions for a given doc (e.g. one terse keyword-style query, one full-sentence question, one from a beginner's phrasing) — don't just reword the doc's own heading. 4 5Respond with ONLY a JSON object of the shape {"entries": [...]}...`
The "don't just reword the doc's own heading" line is doing real work: without it, a first pass produced queries that were near-verbatim titles — training the model to match surface strings instead of meaning, which is exactly the grep-shaped behavior this whole project is trying to get past.
One query becomes one training example
1for _, e := range got { 2 for _, q := range e.Queries { 3 q = strings.TrimSpace(q) 4 if q != "" { 5 pairs = append(pairs, Pair{Query: q, Doc: e.Path}) 6 } 7 } 8}
A doc with 3 generated queries becomes 3 separate (query, doc) rows — flattened,
because the training loss (next section) operates on one query/one positive-doc at a time, not
on a doc with a list of valid queries attached.
1// dropLeakedPairs removes any training pair whose query is a 2// near-duplicate (token Jaccard similarity > 0.7) of a real eval query 3func dropLeakedPairs(pairs []Pair, evalQueries []string) []Pair { 4 for _, p := range pairs { 5 pTokens := tokenize(p.Query) 6 for _, evalTokens := range evalTokenSets { 7 if jaccard(pTokens, evalTokens) > 0.7 { leaked = true 8 } 9 } 10 } 11}
This is the two-tower diagram and the InfoNCE loss from the field guide's §02 and §05, as an
actual training loop. There is surprisingly little code — nearly all of it is
sentence-transformers doing the batching and gradient step; the file mostly just
wires your pairs and your GPU into it.
Every pair becomes one training example
1examples = [ 2 InputExample(texts=[p["query"], doc_text(p["doc"])]) 3 for p in pairs 4]
InputExample(texts=[a, b]) is sentence-transformers' way of saying "these
two texts are the anchor and the positive" — literally the query dot and the green document
dot from the embedding-space diagram, paired up.
1train_loader = DataLoader(examples, shuffle=True, batch_size=args.batch_size) 2loss = losses.MultipleNegativesRankingLoss(model)
This one line is the InfoNCE formula from the field guide:
MultipleNegativesRankingLoss computes exactly this, and it gets its
negatives — the "Σᵢ" over all dᵢ — for free from whatever else happens to be in the
batch. With batch_size=32, every query trains against 31 negatives it never had to
be told about explicitly; double the batch size and every example gets a harder, more informative
training signal at no extra data-collection cost. This is also the number that trades most
directly against your 10GB VRAM budget — bigger batches need more memory.
Running it on the GPU
1model = SentenceTransformer(args.base, device="cuda") 2… 3model.fit( 4 train_objectives=[(train_loader, loss)], 5 epochs=args.epochs, 6 warmup_steps=warmup_steps, 7 output_path=args.out, 8)
This is literally the "step 1 / step 2 / step 3" loop from the training animation —
.fit() just runs it thousands of times: sample a batch, compute the loss above,
take a gradient step that pulls each query closer to its true doc and (implicitly, via the
softmax denominator) away from the other 31 docs in the batch. output_path is where
the nudged weights land — this is the file eval_embed.py loads next.
This is the "compare vectors with a dot product" half of the two-tower diagram, done for all 45 eval queries against all 457 docs at once — the brute-force approach from §07, since 457 × 384 floats has no business needing a vector database.
1doc_vecs = model.encode(doc_texts, batch_size=32, normalize_embeddings=True) 2query_vecs = model.encode(query_texts, batch_size=32, normalize_embeddings=True)
Same split as the bi-encoder diagram: 457 docs through the encoder once (this is the offline pass — in a real deployment this line runs once and gets cached, not on every query), 45 queries through the same encoder, shared weights, separately.
1sims = query_vecs @ doc_vecs.T # (45, 457) cosine similarity 2… 3order = np.argsort(-sims[i]) 4ranked = [doc_paths[j] for j in order[:10]]
Because both sides were normalize_embeddings=True, a plain dot product
is cosine similarity — the @ is the "·" node from the bi-encoder diagram,
just batched into one (45 × 384) @ (384 × 457) multiply instead of 45×457 separate
comparisons. This is also the line that would need to change to FAISS/HNSW if the corpus
ever grew past the point where numpy can do this in microseconds.
The same hit/acceptable rule as the Go evalrun harness
1hit = paths_equal(top1, ideal) 2acc = hit or any(paths_equal(top1, a) for a in acceptable_docs)
Deliberately copied from score() in agent/cmd/evalrun/main.go,
not reinvented — hit requires the exact ideal_doc,
acceptable also allows anything in acceptable_docs. Same rule, same
eval set, means a bi-encoder's score lands on the same scoreboard as the graph agent and
single-shot agent methods already in results/: 23/45 and 26/45 exact hits.