Sprocket is a 501-million-parameter language model trained from scratch on one rented GPU. This page walks through what it is made of and how it got that way, using the real vocabulary and the real numbers from the training run. Everything here runs in your browser.
A model cannot read letters. Something has to turn writing into numbers first.
A language model works with a fixed list of numbered pieces of text. Every input has to be cut into pieces from that list, and each piece gets swapped for its number. The model only ever sees the numbers.
The list is called the vocabulary, and the thing that does the cutting is the tokenizer. Sprocket's vocabulary has 32,000 entries. They are not words. Common words are usually a single entry, but rarer words get split into fragments, and a word the vocabulary has never seen still works because it can always fall back to smaller pieces.
This matters more than it sounds. The vocabulary is fixed before training starts and can never change afterwards, because every number the model learned would point at something different. It also sets the price of everything: a tokenizer that needs more pieces to write the same sentence means more compute per page of text, a shorter effective memory, and a larger bill.
Alternating colours mark where one token ends and the next begins. · stands for a space that belongs to the token after it.
Nobody chose them. The vocabulary was learned from text, using an approach called byte pair encoding, and the idea behind it is simple enough to describe in a sentence: start with the smallest possible pieces, then repeatedly glue together whichever pair shows up next to each other most often.
It starts with raw bytes, so the beginning vocabulary is just 256 entries and
every possible input is already representable. Then it counts pairs across a
large sample of text. If t and h are the most common
adjacent pair, th becomes entry 257. Now it counts again, with
th treated as a single unit, and maybe th and
e merge into the. Repeat until the vocabulary is the
size you asked for.
Sprocket's ran for 31,728 merges over a sample of FineWeb-Edu, the same source used for the main training. The order of those merges is kept, and encoding just replays them: the tokenizer above is applying that exact list to whatever you type, lowest-numbered merge first.
Because the merges were learned from ordinary English web text, that is what the vocabulary is efficient at. It averages about 4.5 characters per token on that kind of writing. Feed it something unlike its training text and the number drops, which is exactly what you see if you paste code or an unusual word into the box above. Nothing breaks. It just takes more pieces, and more pieces cost more.
Working at the byte level is what makes the "nothing breaks" part true. There are far more possible characters in the world than any 32,000-entry list can hold, so a tokenizer built from characters has to decide what to do with the ones it does not know, and the usual answer is a placeholder that destroys the text. Bytes have no such problem: there are only 256 of them, all of them are in the vocabulary from the start, and anything at all can be spelled out.
Not every entry was learned. Sixteen were set aside by hand before training, and they are the only ones that mean something structural rather than textual. They are how the model can tell who is speaking, where a turn ends, and when it is doing something other than replying.
| ID | Token | What it marks |
|---|
Reserving these is a decision made once and permanently. They occupy IDs 0 to 15, which is why the learned vocabulary starts at 16. Adding a new one after training would mean a piece the model has never seen, so the full set has to be planned before the first token is ever encoded.
501 million numbers, arranged in a particular shape. The shape is the design.
Once text is a list of numbers, the model turns each number into a long list of decimals, then pushes that through the same block of machinery 26 times over. At the end it produces a score for every one of the 32,000 vocabulary entries, and the highest score is its guess at what comes next. That is the entire job: guess the next piece, over and over.
Each of those 26 blocks does two things. First, attention, which lets every position look back at the earlier positions and pull in what is relevant, so that by the time the model reaches "it" in a sentence it can carry information about what "it" refers to. Second, a feed-forward step, which is where most of the parameters live and where most of what the model knows is stored.
A parameter is just one adjustable number. Training means nudging all 501 million of them, repeatedly, until the guesses get better. Nobody sets them by hand and nobody can point at one and say what it does.
These are exact counts, not estimates. scripts/verify_web_modelmath.py
builds each real model in PyTorch and fails if this page disagrees by a single
parameter.
Drag width and the count moves fast, because widening the model makes every matrix bigger in both directions at once. Drag layers and it moves in a straight line, because each layer is a fixed cost. Wider models hold more per layer; deeper models do more steps of reasoning. Sprocket is relatively deep and narrow for its size, which suits a model meant to run on small hardware.
Now drag key/value heads and watch how little the parameter count cares. It barely moves, which makes it look unimportant. It is not. That setting controls something the parameter count cannot show you.
While the model is generating, it keeps notes on everything it has already written so it does not have to redo that work for every new word. Those notes grow with every token, and on a phone or a laptop they, not the model itself, are usually what runs the memory out.
Sprocket uses 20 attention heads but only 4 sets of those notes, shared between them. That is a five-fold reduction in the thing that grows without limit, in exchange for a small amount of flexibility. At its full 2,048-token context the notes come to 0. With one set of notes per head, the same context would need 0.
This is the kind of decision that only makes sense if you know where the model is going to run. For something intended to work on a laptop or a phone, shrinking the part that grows with conversation length matters more than the part that is fixed the moment training ends.
Twenty billion tokens, 152,580 adjustments, 54 hours, one measurement of whether it was working.
Training is a loop. Show the model some text with the next piece hidden, let it guess, measure how wrong the guess was, and nudge every parameter a little in the direction that would have made it less wrong. Then do that 152,580 times.
The measure of wrongness is called loss, and it has a meaning you can hold onto. A model that knows nothing at all would spread its guess evenly across all 32,000 entries, and that gives a loss of about 10.4. A model that always guessed correctly with total confidence would score 0. Every real model lives somewhere in between, and the whole run is just that number coming down.
Sprocket started at 10.4, exactly where a model that has never seen text starts, and finished at 2.564. Another way to say the same thing: at the end it was about as uncertain as if it were choosing between roughly 13 options each time, rather than 32,000.
The steep drop at the very start is the model learning that some letters are common and some are rare. It takes minutes and looks impressive on a chart. The remaining 95% of the run is the part that actually produces a usable model, and on a chart it looks like almost nothing happening, which is why the first view above hides the opening plunge.
The long run produces something that continues text but does not answer. Ask it a question and it might reply with more questions, because a great deal of the internet is questions followed by more questions. It has learned language, not conversation.
A second, much shorter stage fixes that, training on example exchanges. The scoring is masked so the model is only graded on the replies, never on predicting what the person said. That distinction matters: without it the model spends capacity learning to imitate users, which is not the job.
That second stage is about 1,200 steps against 152,580. Almost all the compute goes into learning language, and a comparatively tiny amount goes into shaping how it responds. That ratio is roughly what it looks like for every chat model.
Every architecture decision eventually turns into an hourly rate.
Training cost is mostly one calculation: work per token, times number of tokens, divided by how fast the hardware does work. Work per token is set by the parameter count, so a model twice the size takes about twice as long on the same machine. Nothing about this is subtle, but it is easy to not do until the bill arrives.
Scaled from throughput measured on the actual machine, 115,295 tokens per second for this model, not from a hardware vendor's peak figure. It covers the main training stage only.
Slide the token count up and the cost rises in a straight line. Quality does not. Going from 10 billion tokens to 50 billion costs five times as much and buys roughly a 21% improvement, with most of the benefit arriving early. The curve flattens somewhere around 15 to 20 billion, which is why 20 was chosen.
The number that decides what a model should be compared against is tokens per parameter. Sprocket saw 40. GPT-2-medium saw about 28 and Cerebras-GPT-590M saw 20, so those are its honest peer group. Qwen2.5-0.5B is almost exactly the same size but saw around 36,000, which is roughly 900 times more text. It is a far better model, and no architectural cleverness closes a gap like that. Only more training does, and more training is the line going up in the panel above.
Development happened on a desktop graphics card with 8GB of memory, about 7GB of it usable. Running a model needs room for its weights. Training one needs room for the weights, a full set of correction values the same size, and two more sets the optimizer keeps as running averages. That comes to roughly 16 bytes per parameter, so the memory needed to train is about eight times what is needed to run.
For Sprocket that is 8.0GB of pure bookkeeping before a single piece of text is loaded, against 7GB available. It missed, and not by an amount any tuning could recover. Go back to the sliders in section 2 and watch "memory to train" against "memory to run" to see how quickly that gap opens.
So the plan became: build and debug everything against a small model that trains overnight on the desktop, and rent a large GPU only once the identical code had run start to finish unattended. The rented machine is for training, not for finding out that a path was misspelled.