Lab
How this actually works
Four pieces to try out. Everything runs here in your browser, nothing goes to a server — and each one states what is computed and what is merely shown.
Why does a German word cost more than an English one?
A model does not read letters, it reads pieces. It first splits text at word and punctuation boundaries — exactly what happens here, live. Long words then break down further, and German umlauts count double: they need two bytes where an English letter needs one. That is why the same sentence costs more in German.
The·tram·leaves·at·2:30·pm·—·best·regards!
≥ 13 — at least this many tokens
Fully computed. The split uses the very pattern GPT models pre-tokenise with; characters and bytes are counted by your browser. Only the final merging into tokens is missing — which is why the number is a lower bound, not an estimate.
How can a machine know that two words are related?
By turning meaning into a place. Every word gets a position, and what is similar sits close together. A search then no longer asks "does this word appear", but "what lies nearby". That is exactly why a good search finds "invoice" when you type "receipt".
- 1. receiptDistance 0.106
- 2. billDistance 0.108
- 3. paymentDistance 0.144
The computation is real: neighbours are found by distance, exactly as a vector database does it. The map itself was laid out by hand. A real model works in hundreds of dimensions and learns the positions — that would be a few hundred megabytes of download for a demonstration.
How does the model know what "it" refers to?
Every word looks at every other and assigns weights. "It" looks at "invoice" and finds more there than at "yesterday" — and that weighting is the actual invention behind every language model in use today. Pick a word: what lights up is what it listens to.
Click a wordsoftmax(q·k / √d)Weight: invoice
The formula is the real one — softmax(q·k over the square root of d), computed word by word. The vectors were set by hand so you can see what they do; a model learns them from billions of sentences. This is one attention rather than many, and the numbers are coarse — the mechanism is exactly this.
What does it really cost to run a model yourself?
For every single word, a model has to be read out of memory in full. That is the bottleneck — not compute. A model half the size is therefore twice as fast, and a machine with fast memory beats one with a fast CPU. Move the sliders and see where the limit sits.
200 GB/s ÷ 8 GB = 25.0 · Assumed: 60 words per second, 40 ms network round trip per request
Real arithmetic on a deliberately simple assumption: words per second equals memory bandwidth divided by model size. That is the accepted upper bound and sits 20 to 40 per cent above practice. The cloud figures are a fixed assumption, stated below — real providers vary through the day.
What we are thinking about
Three questions we do not have a good answer to yet. We write them down because an honest open question is worth more than a closed one nobody asked.
- measuring
How much context is too much?
A local model can take in an entire manual today. It does not get smarter for it, though — it gets slower and blurrier, pulling its answer towards the start and the end of the text while leaving the middle alone. We are looking for the point where targeted lookup beats handing over everything.
- built, unfinished
How do you prove what an assistant did?
When an assistant approves an invoice, that has to be traceable — not as a text log, but as a chain that still holds a year later. In A2 Apex we record every tool call. What is missing is the form in which an auditor can read it without asking us.
- unsolved
What does an answer really cost?
Providers bill in tokens, and you only learn those afterwards. In A2 Apex we count along ourselves so we can estimate up front. Own runtimes are the more interesting case: there the honest unit is not the token but the occupied second on a GPU — and nobody has a good price for that yet.