The mining loop is short on purpose: it's the exact same encode-then-argsort shape as
eval_embed.py, run against training queries instead of eval queries. That's not a
coincidence — mining is retrieval. The question "what would this model currently return
for this query" is the same question whether you're scoring it or about to correct it.
1doc_vecs = model.encode(doc_texts, batch_size=32, show_progress_bar=True, normalize_embeddings=True) 2query_vecs = model.encode([args.query_prefix + p["query"] for p in pairs], batch_size=32, show_progress_bar=True, normalize_embeddings=True) 3 4sims = query_vecs @ doc_vecs.T # (num_pairs, num_docs) cosine similarity
Same brute-force matrix multiply as the eval script (here the matrix is 1,372 × 457) — 1,372 training queries
against 457 docs is a bigger matrix than eval's 45×457, but still nothing a GPU notices. This
line is why mining has to run per base model: model.encode is base-model-specific,
so sims is a different 1,372×457 matrix for MiniLM than for bge-small. "Confusable
with the right answer" is a fact about a specific model's pretraining, not a fact about the corpus.
1for i, p in enumerate(pairs): 2 order = np.argsort(-sims[i]) 3 hard_negatives = [] 4 for j in order: 5 candidate = doc_paths[j] 6 if paths_equal(candidate, p["doc"]): 7 continue 8 hard_negatives.append(candidate) 9 if len(hard_negatives) >= args.top_k: 10 break
Walk the ranked list nearest-first, and take the first top_k (2, here)
that aren't the pair's own true document. That's the whole mining step. Whatever survives
this filter is, by construction, the model's own best guess at "which wrong document looks most
like the right one" — literally the H vector from the loss writeup, computed for
real instead of assumed.
eval/ground_truth.json tracks this
distinction for the 45 eval queries via acceptable_docs — but that oracle doesn't exist
for the 1,372 synthetic training queries. There's no code fix for this here; some mined pairs
will carry false negatives, same caveat "The Best Wrong Answer" raised before this script
existed.
Both changes plug directly into InputExample and MultipleNegativesRankingLoss,
which already had the hooks for exactly this — nothing about the training loop itself needed to
change.
1texts = [args.query_prefix + p["query"], doc_text(p["doc"])] 2if args.num_hard_negatives > 0: 3 for neg in p.get("hard_negatives", [])[:args.num_hard_negatives]: 4 texts.append(doc_text(neg)) 5examples.append(InputExample(texts=texts))
InputExample(texts=[query, positive]) was the whole story before; now
it can be [query, positive, hard_neg_1, hard_neg_2]. Nothing downstream needed to
change to make this work — MultipleNegativesRankingLoss already treats every text
past index 1 as an additional negative, on top of whatever the rest of the batch provides for
free. The library was built for this; the mining script is what was missing.
1loss = losses.MultipleNegativesRankingLoss(model, scale=args.scale)
One word added to one line. scale defaults to 20 in the library, and
the entire September checkpoint used that default for all three base models without ever
touching it. This sweep is the first time it's actually a variable instead of an assumption.
The first sweep pass crashed here, mid-run, for bge-small and e5-small
specifically — never for MiniLM. This is CUDA (the GPU's compute layer) refusing to allocate more memory than the card physically has:
# reconstructed from the run log; the exact numbers weren't saved 1torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 72.00 MiB. 2GPU 0 has a total capacity of 9.62 GiB of which 140.31 MiB is free.
Two extra hard-negative texts per example doubles how many texts one training batch has
to embed at once (2 → 4), and triples the number of long, document-length ones (1 → 3). Two things then separate MiniLM from the other two: it is 6 transformer layers
where bge-small and e5-small are 12, and its
max_seq_length is 256 where theirs is 512 (e5 too) — and since
train_biencoder.py feeds 2,000-character excerpts that tokenize to ~500 pieces for most
docs, the 12-layer models are also attending over up to twice as many tokens per text, with
attention cost growing with the square of that length. Twice the depth times up to twice the
sequence length, for a batch carrying three times as many document-length texts. batch_size=32 stopped fitting; dropping to 16 still didn't for the two
12-layer models, and it took dropping to 6 before training would run to
completion.
batch_size also sets the
negative count, and MultipleNegativesRankingLoss pools every example's mined
negatives into one shared candidate set, so each query is scored against
batch × (1 + mined) − 1 candidates. MiniLM's hard-negative runs went from 31
negatives/example (batch 32, baseline) to 47 (batch 16: 15 other positives + 32
mined docs from across the batch) — more negatives, not fewer. bge-small's and
e5-small's went from 31 down to 17 (batch 6: 5 other positives + 12
mined). So for those two models "mined vs. random negatives" is confounded with "17 negatives vs.
31," plus a 5× smaller batch and correspondingly noisier, more numerous gradient steps —
all forced by VRAM, not chosen as part of the experiment. MiniLM's row pair is the closest thing
to an apples-to-apples test, and even it changes two things at once (mined negatives, and 47 vs. 31).
coverage: none, which a bi-encoder can never get right because it cannot abstain) · hit and acceptable are defined on the pipeline page| model | scale | hard negs | batch | exact@1 | acceptable@1 | recall@5 |
|---|---|---|---|---|---|---|
| MiniLM | 8 | 0 | 32 | 37.8% | 51.1% | 82.5% |
| MiniLM | 8 | 2 | 16 | 37.8% | 53.3% | 85.0% |
| MiniLM | 20 | 0 | 32 | 35.6% | 44.4% | 87.5% |
| MiniLM | 20 | 2 | 16 | 26.7% | 44.4% | 80.0% |
| bge-small | 8 | 0 | 32 | 33.3% | 42.2% | 77.5% |
| bge-small | 8 | 2 | 6 | 31.1% | 40.0% | 77.5% |
| bge-small | 20 | 0 | 32 | 33.3% | 42.2% | 75.0% |
| bge-small | 20 | 2 | 6 | 28.9% | 40.0% | 70.0% |
| e5-small | 8 | 0 | 32 | 28.9% | 40.0% | 82.5% |
| e5-small | 8 | 2 | 6 | 33.3% | 42.2% | 75.0% |
| e5-small | 20 | 0 | 32 | 35.6% | 44.4% | 82.5% |
| e5-small | 20 | 2 | 6 | 31.1% | 42.2% | 77.5% |
Highlighted rows: best config per model, by recall@5 first, acceptable@1 to break ties. One run per configuration — no fixed random seed, so single-point gaps are noise, not signal. That also means the scale-20 / no-negatives rows here are fresh runs of the same recipe as the September checkpoint, not the checkpoint itself: MiniLM landed at 35.6% / 44.4% / 87.5% this time versus the published 40.0% / 48.9% / 92.5%, which is the size of run-to-run variance to keep in mind when reading every other gap in this table.
MiniLM: the closest thing to a win, and why it doesn't count
At scale 8, adding two mined negatives moved MiniLM up by one query on acceptable@1 and one on recall@5, and tied it on exact@1. At scale 20 the same two negatives cost it four queries on exact@1 and three on recall@5 (acceptable@1 tied). That +1/+1 is the only hard-negative gain in the sweep that came without a loss on another metric (e5-small at scale 8 also gained on both @1 metrics, but gave back three queries on recall@5) — and it is smaller than the run-to-run spread the table caption just quoted, and no bigger than the one-query recall@5 gap to MiniLM's own scale-20 / no-negatives row, which by the stated selection rule (recall@5 first) is the best MiniLM configuration here even though the scale-8 hard-negative row beats it on both @1 metrics. MiniLM was the model hard negatives were hypothesized to help most, being the weakest base model with the most room to move; a single unseeded run can't say whether they did.
bge-small and e5-small: hard negatives never won by the selection rule
Both models' best rows by recall@5 are in-batch-only. That isn't quite "hard negatives
lost on every metric": bge-small's hard-negative rows are at or below their baselines everywhere,
but e5-small's scale-8 hard-negative row is two queries up on exact@1 and one up on
acceptable@1, while three down on recall@5. Read alongside §03's confound, the honest conclusion
isn't "hard negatives don't help deeper models" — it's "this sweep couldn't isolate whether they do,
because the VRAM-forced batch-size cut changed the negative count and the optimization dynamics at
the same time." A re-run with CachedMultipleNegativesRankingLoss, which
keeps the full in-batch negative count at a fraction of the memory via GradCache (plain gradient accumulation
wouldn't — it only accumulates gradients, it doesn't put the missing negatives back in the
softmax), would actually test it
deferred
One number moved independently of hard negatives, though: bge-small's best two rows are both scale 8, never scale 20 — mild, not dramatic, support for the earlier hypothesis that its already-well-calibrated pretrained space wants a gentler dial than MiniLM's does.
Check your understanding
1. Why does mine_hard_negatives.py need to run separately for each base model rather than once for the whole corpus?
sims = query_vecs @ doc_vecs.T is a different 1,372×457 matrix for MiniLM than for bge-small, because model.encode is base-model-specific. "Confusable with the right answer" depends on that specific model's own pretraining.
2. What real risk does mining hard negatives from nearest-neighbor similarity introduce, one the eval set's acceptable_docs field guards against for eval queries but not for training queries?
The callout in §01 says this directly: there's no oracle like acceptable_docs for the 1,372 synthetic training queries, so some mined pairs will carry false negatives, with no code fix available.
3. Why did appending hard-negative texts to InputExample's text list work with zero changes to MultipleNegativesRankingLoss's own code?
The library was built for exactly this — InputExample(texts=[query, positive, neg1, neg2]) needed no downstream change because the loss already sums over every candidate past the positive.
4. Why is the hard-negative comparison for bge-small and e5-small called "confounded," while MiniLM's isn't?
Because the loss pools every example's mined negatives, MiniLM's batch-16 runs actually had 47 negatives per query (15 other positives + 32 mined); bge-small/e5-small's batch-6 runs had 17 (5 + 12), down from 31 — a VRAM-forced cut that buries "mined vs. random" under "fewer negatives, tiny batch."
5. What pattern in the sweep suggests loss scale (temperature) matters somewhat independently of hard negatives?
Both of bge-small's best rows land at scale 8 — mild but consistent support for the idea that an already-well-calibrated embedding space wants a gentler dial than a weaker model does.