Inside the transformer

A transformer never sees the string "reset workspace config". Before anything else, a tokenizer — a separate, much simpler piece of software with a fixed vocabulary (30,522 entries for MiniLM's, inherited from BERT) — chops the string into subword pieces and looks each one up as an integer.

Each token id indexes a row in an embedding table — a 30,522 × 384 matrix of learned numbers, one 384-dimensional row per vocabulary entry. Because a transformer has no built-in sense of word order (unlike an RNN, which reads one token at a time), a second vector — the positional embedding, one per position 0, 1, 2, … — is added on top, so "config workspace" and "workspace config" don't look identical to the model.

Every token in the input ends up as one such 384-number vector. A 6-word query becomes a 6×384 matrix — this is what actually enters the first transformer block.

This is the mechanism that made transformers replace RNNs: every token's vector gets updated by looking at every other token in the sequence at once, weighted by how relevant each one is. Each token produces three projections of itself — a query (what am I looking for), a key (what do I offer), and a value (what do I actually contribute) — all three are just learned matrix multiplications of the same input vector.

Read it as: score every token's key against my query (QKT), scale it down so the softmax doesn't saturate (/√d), turn the scores into weights that sum to 1 (softmax), then take a weighted average of everyone's value vectors. "Reset" ends up attending heavily to "config" and "workspace" — the softmax literally learns that those tokens matter for disambiguating what "reset" means here.

Why 12 heads instead of 1

A single attention pass can only learn one notion of "relevant." Running 12 of them in parallel — each with its own learned Q/K/V projections, on a 32-dimensional slice of the 384 — lets different heads specialize: one might track subject-verb agreement, another might link a pronoun back to its noun, another might just look at adjacent tokens. Their 12 outputs are concatenated back to 384 dimensions and passed through one more learned projection.

The full block, and stacking six of them

Multi-head attention is one half of a transformer block. The other half is a small feedforward network (384 → 1536 → 384, with a nonlinearity in between) applied to every token independently — attention mixes information across tokens, the feedforward layer processes each token's mixed result on its own. Both halves get a residual connection (add the input back to the output) and a layer normalization, which is what makes it possible to stack these blocks deeply without training collapsing.

After 6 blocks, a 6-token query is still 6 separate 384-dim vectors — one per token, now richly contextualized. But the bi-encoder from the first field guide needs exactly one vector per query and one per document to take a dot product. Pooling is the step that collapses the many into the one, and it's the one sentence-transformers-specific idea in this whole page — a base BERT model doesn't do this on its own.

Everything above — tokens, embeddings, attention, blocks, pooling — is shared machinery. The one structural difference between an embedding model like MiniLM and a chat model like Claude or GPT is which tokens are allowed to attend to which.

Steps 1–3 above are where a model actually learns language and general-purpose meaning. Step 4 — the only one you've run — assumes all of that already exists and just steers it. Training a model from nothing directly on the deno-docs corpus means skipping straight to step 4 with no steps 1–3 underneath it: there's no pretrained space to steer, because nothing has taught the model what a sentence is yet.

That's roughly a 10,000× gap. It isn't just "less data, worse model" — below a certain scale, a transformer trained from random initial weights doesn't reliably learn grammar, word relationships, or general semantics at all; it just memorizes surface patterns in the 350K words it saw. That's why every field-guide result so far started from a pretrained checkpoint and only fine-tuned: the pretraining is what makes the space worth steering, and no static 2,000-document corpus is anywhere near enough text to grow one from nothing.