Inside the transformer

A transformer never sees the string "resetting file permission". Before anything else, a tokenizer — a separate, much simpler piece of software with a fixed vocabulary (30,522 entries for MiniLM's, inherited from BERT) — chops the string into subword pieces and looks each one up as an integer.

Each token id indexes a row in an embedding table — a 30,522 × 384 matrix of learned numbers, one 384-dimensional row per vocabulary entry. Because a transformer has no built-in sense of word order (unlike an RNN, which reads one token at a time), a second vector — the positional embedding, one per position 0, 1, 2, … (learned, BERT-style, not the sinusoids of the original Transformer) — is added on top, so "permission file" and "file permission" don't look identical to the model.

Every token in the input ends up as one such 384-number vector. The four pieces above, plus the [CLS] and [SEP] markers the tokenizer wraps every input in, make a 6×384 matrix — this is what actually enters the first transformer block.

This is the mechanism that the Transformer paper introduced in place of recurrence: every token's vector gets updated by looking at every other token in the sequence at once, weighted by how relevant each one is. Each token produces three projections of itself — a query (what am I looking for), a key (what do I offer), and a value (what do I actually contribute) — all three are just learned matrix multiplications of the same input vector.

Read it as: score every token's key against my query (QKT), scale it down so the softmax doesn't saturate (/√d), turn the scores into weights that sum to 1 (softmax), then take a weighted average of everyone's value vectors. "Reset" ends up attending heavily to "file" and "permission" — the learned Q/K projections are what make those tokens matter for disambiguating what "reset" means here.

Why 12 heads instead of 1

A single attention pass can only learn one notion of "relevant." Running 12 of them in parallel — each with its own learned Q/K/V projections, on a 32-dimensional slice of the 384 — lets different heads specialize: one might track subject-verb agreement, another might link a pronoun back to its noun, another might just look at adjacent tokens. Their 12 outputs are concatenated back to 384 dimensions and passed through one more learned projection.

The full block, and stacking six of them

Multi-head attention is one half of a transformer block. The other half is a small feedforward network (384 → 1536 → 384, with a nonlinearity in between) applied to every token independently — attention mixes information across tokens, the feedforward layer processes each token's mixed result on its own. Both halves get a residual connection (add the input back to the output) and a layer normalization; the residual path is what makes it possible to stack these blocks deeply without training collapsing.

After 6 blocks, a 6-token query is still 6 separate 384-dim vectors — one per token, now richly contextualized. But the bi-encoder from the first field guide needs exactly one vector per query and one per document to take a dot product. Pooling is the step that collapses the many into the one, and it's the one step here that sentence-transformers added on top of BERT for retrieval — base BERT has a [CLS] pooler, but it was never trained to make that vector comparable across sentences.

Everything above — tokens, embeddings, attention, blocks, pooling — is shared machinery. The defining structural difference between an embedding model like MiniLM and a GPT-style chat model (Claude's internals aren't published; the comparison here is to the published GPT-style decoders) is which tokens are allowed to attend to which.

Steps 1–3 above are where a model actually learns language and general-purpose meaning. Step 4 — the only one you've run — assumes all of that already exists and just steers it. Training a model from nothing directly on the deno-docs corpus means skipping straight to step 4 with no steps 1–3 underneath it: there's no pretrained space to steer, because nothing has taught the model what a sentence is yet.

That's roughly a 10,000× gap. It isn't just "less data, worse model" — below a certain scale (roughly 10–100 million words, by one careful study), a transformer trained from random initial weights learns grammar, word relationships, and general semantics far less reliably, and leans on surface patterns in the 330K words it saw. That's why every field-guide result so far started from a pretrained checkpoint and only fine-tuned: the pretraining is what makes the space worth steering, and no static 2,000-document corpus is anywhere near enough text to grow one from nothing.