The mining loop is short on purpose: it's the exact same encode-then-argsort shape as
eval_embed.py, run against training queries instead of eval queries. That's not a
coincidence — mining is retrieval. The question "what would this model currently return
for this query" is the same question whether you're scoring it or about to correct it.
1doc_vecs = model.encode(doc_texts, batch_size=32, normalize_embeddings=True) 2query_vecs = model.encode([p["query"] for p in pairs], batch_size=32, normalize_embeddings=True) 3 4sims = query_vecs @ doc_vecs.T # (1372, 457) cosine similarity
Same brute-force matrix multiply as the eval script — 1,372 training queries
against 457 docs is a bigger matrix than eval's 45×457, but still nothing a GPU notices. This
line is why mining has to run per base model: model.encode is base-model-specific,
so sims is a different 1,372×457 matrix for MiniLM than for bge-small. "Confusable
with the right answer" is a fact about a specific model's pretraining, not a fact about the corpus.
1for i, p in enumerate(pairs): 2 order = np.argsort(-sims[i]) 3 hard_negatives = [] 4 for j in order: 5 candidate = doc_paths[j] 6 if paths_equal(candidate, p["doc"]): 7 continue 8 hard_negatives.append(candidate) 9 if len(hard_negatives) >= args.top_k: 10 break
Walk the ranked list nearest-first, and take the first top_k (2, here)
that aren't the pair's own true document. That's the whole mining step. Whatever survives
this filter is, by construction, the model's own best guess at "which wrong document looks most
like the right one" — literally the H vector from the loss writeup, computed for
real instead of assumed.
eval/ground_truth.json tracks this
distinction for the 45 eval queries via acceptable_docs — but that oracle doesn't exist
for the 1,372 synthetic training queries. There's no code fix for this here; some false negatives
are baked into every mined pair, same caveat "The Best Wrong Answer" raised before this script
existed.
Both changes plug directly into InputExample and MultipleNegativesRankingLoss,
which already had the hooks for exactly this — nothing about the training loop itself needed to
change.
1texts = [args.query_prefix + p["query"], doc_text(p["doc"])] 2if args.num_hard_negatives > 0: 3 for neg in p.get("hard_negatives", [])[:args.num_hard_negatives]: 4 texts.append(doc_text(neg)) 5examples.append(InputExample(texts=texts))
InputExample(texts=[query, positive]) was the whole story before; now
it can be [query, positive, hard_neg_1, hard_neg_2]. Nothing downstream needed to
change to make this work — MultipleNegativesRankingLoss already treats every text
past index 1 as an additional negative, on top of whatever the rest of the batch provides for
free. The library was built for this; the mining script is what was missing.
1loss = losses.MultipleNegativesRankingLoss(model, scale=args.scale)
One word added to one line. scale defaults to 20 in the library, and
the entire September checkpoint used that default for all three base models without ever
touching it. This sweep is the first time it's actually a variable instead of an assumption.
The first sweep pass crashed here, mid-run, for bge-small and e5-small
specifically — never for MiniLM. This is CUDA (the GPU's compute layer) refusing to allocate more memory than the card physically has:
1torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 72.00 MiB.
2GPU 0 has a total capacity of 9.62 GiB of which 140.31 MiB is free.
Two extra hard-negative texts per example roughly doubles how many texts one training batch has
to embed at once. MiniLM is 6 transformer layers; bge-small and e5-small
are 12 — twice the activation memory per text, for a batch that was already about to carry twice as
many texts. batch_size=32 stopped fitting; dropping to 16 still didn't for the two
12-layer models, and it took dropping to 6 before training would run to
completion.
batch_size is also the
in-batch negative count. MiniLM's hard-negative runs went from 31 negatives/example (batch 32,
baseline) to 17 (batch 16: 15 in-batch + 2 mined) — a modest cut. bge-small's and
e5-small's hard-negative runs went from 31 negatives/example all the way down to
7 (batch 6: 5 in-batch + 2 mined). That's not a clean "mined vs. random negatives"
comparison for those two models anymore — it's confounded with "7 negatives vs. 31," a cut forced by
VRAM, not chosen as part of the experiment. Only MiniLM's row pair below is close to an apples-to-apples
test of hard negatives on their own.
| model | scale | hard negs | batch | exact@1 | acceptable@1 | recall@5 |
|---|---|---|---|---|---|---|
| MiniLM | 8 | 0 | 32 | 37.8% | 51.1% | 82.5% |
| MiniLM | 8 | 2 | 16 | 37.8% | 53.3% | 85.0% |
| MiniLM | 20 | 0 | 32 | 35.6% | 44.4% | 87.5% |
| MiniLM | 20 | 2 | 16 | 26.7% | 44.4% | 80.0% |
| bge-small | 8 | 0 | 32 | 33.3% | 42.2% | 77.5% |
| bge-small | 8 | 2 | 6 | 31.1% | 40.0% | 77.5% |
| bge-small | 20 | 0 | 32 | 33.3% | 42.2% | 75.0% |
| bge-small | 20 | 2 | 6 | 28.9% | 40.0% | 70.0% |
| e5-small | 8 | 0 | 32 | 28.9% | 40.0% | 82.5% |
| e5-small | 8 | 2 | 6 | 33.3% | 42.2% | 75.0% |
| e5-small | 20 | 0 | 32 | 35.6% | 44.4% | 82.5% |
| e5-small | 20 | 2 | 6 | 31.1% | 42.2% | 77.5% |
Highlighted rows: best config per model, by recall@5 first, acceptable@1 to break ties. One run per configuration — no fixed random seed, so single-point gaps are noise, not signal.
MiniLM: the one clean win
Both scale-8 rows for MiniLM improved over their in-batch-only baseline, on every metric that moved at all. It's also the model this was hypothesized to help most — the weakest base model of the three, with the most room for a large gradient on a modest amount of signal to actually help.
bge-small and e5-small: hard negatives never won
Every hard-negative row for these two models scores at or below its matching in-batch-only row, on every metric. Read alongside §03's confound, the honest conclusion isn't "hard negatives don't help deeper models" — it's "this sweep couldn't isolate whether they do, because the VRAM-forced batch-size cut buried the signal under a much bigger negative-count cut." deferred a re-run with gradient accumulation, so effective negative count doesn't have to collapse to fit 2 mined negatives in memory, would actually test it.
One number moved independently of hard negatives, though: bge-small's best two rows are both scale 8, never scale 20 — mild, not dramatic, support for the earlier hypothesis that its already-well-calibrated pretrained space wants a gentler dial than MiniLM's does.
Check your understanding— optional, 5 questions
1. Why does mine_hard_negatives.py need to run separately for each base model rather than once for the whole corpus?
sims = query_vecs @ doc_vecs.T is a different 1,372×457 matrix for MiniLM than for bge-small, because model.encode is base-model-specific. "Confusable with the right answer" depends on that specific model's own pretraining.
2. What real risk does mining hard negatives from nearest-neighbor similarity introduce, one the eval set's acceptable_docs field guards against for eval queries but not for training queries?
The callout in §01 says this directly: there's no oracle like acceptable_docs for the 1,372 synthetic training queries, so some false negatives are baked into every mined pair with no code fix available.
3. Why did appending hard-negative texts to InputExample's text list work with zero changes to MultipleNegativesRankingLoss's own code?
The library was built for exactly this — InputExample(texts=[query, positive, neg1, neg2]) needed no downstream change because the loss already sums over every candidate past the positive.
4. Why is the hard-negative comparison for bge-small and e5-small called "confounded," while MiniLM's isn't?
MiniLM's batch only dropped 32→16 (31→17 negatives) to fit hard negatives; bge-small/e5-small dropped all the way to 6 (31→7 negatives) — a much bigger, VRAM-forced cut that buries "mined vs. random" under "few negatives vs. many."
5. What pattern in the sweep suggests loss scale (temperature) matters somewhat independently of hard negatives?
Both of bge-small's best rows land at scale 8 — mild but consistent support for the idea that an already-well-calibrated embedding space wants a gentler dial than a weaker model does.