Views: 1
Every language model starts with a step that rarely gets any attention. Before a single neuron fires, the text you typed is chopped into pieces, and each piece is swapped for a number. The tool that does the chopping is called a tokenizer, and the most popular way to build one is an algorithm named byte pair encoding, or BPE.
In this guide we build a BPE tokenizer from scratch in Python, and we do it three times: once with plain loops, once with pure functions, and once as a class. Then we give the same job to the Hugging Face tokenizers library and compare. At the end we tokenize a real dataset of more than two million short stories. This is the first stage of a project where we are building a small language model from the ground up, and every number below comes from code we actually ran.
All you need is basic Python. No machine learning background required.
What a tokenizer actually does
A neural network can only do arithmetic. It cannot read “Once upon a time”. So we need a rule that turns text into a list of integers and back again, without losing anything on the way.
Each integer is the ID of a token, which is a chunk of text. The full list of tokens the model knows is its vocabulary. The tokenizer is the tool that converts between text and IDs.
You could pick whole words as tokens, but then every new or misspelled word becomes a problem, and the vocabulary grows huge. You could pick single characters, but then every sentence turns into a very long list, and long lists are expensive for the model. BPE sits in the middle. It starts from the smallest possible pieces and learns, from your data, which pieces are worth gluing together.
Step 1: Turn text into bytes
The smallest pieces we can use are bytes. Every piece of text in every language can be written as bytes, which means the tokenizer will never meet a character it doesn’t know.
text = 'low lower lowest café'
raw = text.encode('utf-8')
ids = list(raw)
print(raw)
print(ids)
print(len(text), len(ids))
The string has 21 characters but the list has 22 numbers. The reason is the last letter. Plain English letters take one byte in UTF-8, but “é” takes two: 195 and 169. Neither number means anything alone. Together they spell “é”.
You may wonder why we don’t use Python’s ord() function, which gives one number per character. The catch is size. Unicode defines more than a million possible characters, so a vocabulary built on them would be enormous, and unseen characters would still be a problem. Bytes only range from 0 to 255, so we start with exactly 256 tokens and cover everything. BPE then fixes the length problem by merging.
How BPE learns, in plain English
The whole algorithm fits in four steps.
- Count how often each pair of neighbouring tokens appears.
- Take the most frequent pair and give it a brand new token ID.
- Replace every occurrence of that pair with the new ID.
- Repeat until you reach the vocabulary size you want.
On our toy sentence, the pair “l” and “o” appears three times, so it becomes token 256. Next, 256 joined with “w” becomes 257, which stands for “low”. After that a space joined with “low” becomes 258, and so on. Frequent words slowly turn into single tokens, and rare words stay split into smaller pieces.
Version one: the imperative way
Imperative code is the most direct style. We keep variables, loop over them, and change them as we go. Here is one round of counting:
pair_counts = {}
for i in range(len(ids) - 1):
pair = (ids[i], ids[i + 1])
pair_counts[pair] = pair_counts.get(pair, 0) + 1
The loop stops one short of the end because the last ID has no neighbour on its right. The call pair_counts.get(pair, 0) returns the current count, or zero if we haven’t seen the pair yet. Then we pick the winner and merge it:
best_pair = max(pair_counts, key=pair_counts.get)
new_id = 256
new_ids = []
i = 0
while i < len(ids):
if i < len(ids) - 1 and (ids[i], ids[i + 1]) == best_pair:
new_ids.append(new_id)
i += 2
else:
new_ids.append(ids[i])
i += 1
We use a while loop here because on a match we jump forward by two positions. One detail worth noticing: in our sentence the pairs “lo” and “ow” are tied at three occurrences. The max function returns the first one it meets, so ties are settled by whichever pair appeared first in the text. Real tokenizers have to define a tie rule too.
Wrap those steps in an outer loop, record every merge in a dictionary, and stop when the best pair appears fewer than twice. Merging a pair seen only once saves almost nothing. Our toy run learned four merges:
- (108, 111) became 256, which is “lo”
- (256, 119) became 257, which is “low”
- (32, 257) became 258, which is “ low” with a leading space
- (258, 101) became 259, which is “ lowe”
The merge dictionary is the entire trained tokenizer. Everything else can be rebuilt from it.
Decoding and encoding
To go back from IDs to text, we build a table that says which bytes each ID stands for. The first 256 IDs are single bytes. Each new ID is the bytes of its two parents joined together:
vocab = {}
for i in range(256):
vocab[i] = bytes([i])
for (a, b), new_id in merges.items():
vocab[new_id] = vocab[a] + vocab[b]
decoded = b''.join(vocab[i] for i in ids).decode('utf-8', errors='replace')
Notice that we decode to text only once, at the very end. If you decoded token by token, a character like “é” would break, because its two bytes would be separated. The errors='replace' option prints a placeholder instead of crashing when the bytes are cut in the middle of a character.
Encoding new text uses the same merge routine, but with no counting. We simply apply the learned merges in the order they were learned:
tokens = list('lowest slow'.encode('utf-8'))
for pair, new_id in merges.items():
new_tokens = []
i = 0
while i < len(tokens):
if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == pair:
new_tokens.append(new_id)
i += 2
else:
new_tokens.append(tokens[i])
i += 1
tokens = new_tokens
The order matters because later merges are built from earlier ones. Token 257 only exists after token 256 has been created. When we tried the merges in reverse order, the tokenizer still ran without any error, but it produced 9 tokens instead of 7. It silently got worse, which is exactly the kind of bug that is hard to spot.
Version two: the functional way
Functional style means small, pure functions. A pure function takes everything it needs as arguments, returns a new value, and never changes anything outside itself. Here are our two building blocks:
from collections import Counter
def get_pair_counts(ids):
return Counter(zip(ids, ids[1:]))
def merge_pair(ids, pair, new_id):
out = []
i = 0
while i < len(ids):
if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
out.append(new_id)
i += 2
else:
out.append(ids[i])
i += 1
return out
The zip(ids, ids[1:]) trick pairs every ID with the one after it, so the index arithmetic disappears. After calling merge_pair, the original list is untouched, which makes the functions easy to test and safe to call twice. With these two pieces, training becomes a short function, and encoding can even be written as a fold with functools.reduce, where the list of IDs is the value that gets updated once per merge.
Version three: the object oriented way
The third style groups the data and the behaviour into a class. The tokenizer now owns its merge table and its vocabulary, so you never pass them around by hand:
class BPETokenizer:
def __init__(self):
self.merges = {}
self.vocab = {i: bytes([i]) for i in range(256)}
def train(self, text, num_merges, min_count=2):
ids = list(text.encode('utf-8'))
for _ in range(num_merges):
counts = Counter(zip(ids, ids[1:]))
if not counts:
break
best = max(counts, key=counts.get)
if counts[best] < min_count:
break
new_id = 256 + len(self.merges)
ids = merge_pair(ids, best, new_id)
self.merges[best] = new_id
self.vocab[new_id] = self.vocab[best[0]] + self.vocab[best[1]]
Add encode and decode methods and you have a complete tokenizer. We also added a stats method that reports characters, bytes, tokens and the compression ratio, which is bytes divided by tokens. A ratio of 1.0 means no compression at all. This number becomes our scoreboard for the rest of the project.
Writing the same algorithm three ways is not busywork. Each style shows a different side of the idea. The loops show what happens, the functions show what depends on what, and the class shows what belongs together.
Trying it on real text
A toy sentence proves very little, so we trained on 300 stories from the TinyStories dataset, about 258,000 characters, and learned 300 merges. It worked, but slowly. The first run took 47.8 seconds, and a later run took 19.6 seconds. The compression ratio was 2.532 on the training text and 2.557 on 50 stories the tokenizer had never seen.
The first tokens it learned were revealing: “e ”, “d ”, “ t”, “. ”, “he ” and “ the ”. Many of them end with a space or cross a word boundary, because our tokenizer treats the text as one long stream of bytes and has no idea where words begin or end.
Speed is the bigger problem. Our version rescans the whole text for every merge. The full dataset is roughly 8,000 times larger, and we want about 7,900 merges, not 300. Doing that arithmetic gives months of waiting, which is why real projects use a library.
Switching to the Hugging Face tokenizers library
The tokenizers library does the same job, written in Rust. This is the setup:
from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tokenizer.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(
vocab_size=556,
initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
)
tokenizer.train_from_iterator(stories, trainer=trainer)
On the same 300 stories and the same 556 token vocabulary, training took 0.08 seconds instead of 19.6 or 47.8. That is more than 200 times faster. The compression was almost identical: 2.542 against our 2.557.
If compression is about the same, what do we gain? Quality. The library adds a step called pre-tokenization. Before BPE runs, it splits the text into word sized chunks with a regular expression, so no merge can ever cross a word boundary. The space also attaches to the front of the next word instead of the back of the last one. That is why the library learned tokens like “Ġthe”, “Ġand” and “Ġto”, where “Ġ” is simply how byte level tokenizers display a space. Nine of our first twenty tokens ended in a space, and those are weak units for a model to learn meaning from.
Pre-tokenization also makes training faster, because identical words only need to be counted once.
Choosing the vocabulary size
With the library, a realistic vocabulary is cheap to train. We used 20,000 stories and set the size to 8,192. Training took about three seconds. On held out stories, compression jumped from about 2.5 to about 4.2 bytes per token, so the same text needed roughly 40 percent fewer tokens.
Notice the diminishing returns, though. The vocabulary grew fifteen times, but compression improved only about 1.7 times. Every extra token adds a row to the model’s embedding table, so bigger is not free. Vocabulary size is a trade between shorter sequences and a heavier model.
We also registered a special token, <|endoftext|>. We place it after every story so the model learns where one story ends and an unrelated one begins. It lands at ID 0, the 256 byte symbols take IDs 1 to 256, and the first learned merge is ID 257.
Tokenizing the whole dataset
The last step is turning the full dataset into numbers on disk. We loaded the TinyStories train and validation splits, tokenized them in batches of 1,000 stories with encode_batch, and appended each batch to a single binary file:
with open('train.bin', 'wb') as f:
for start in range(0, len(ds), 1000):
texts = ds[start:start + 1000]['text']
chunk = []
for enc in tokenizer.encode_batch(texts):
chunk.extend(enc.ids)
chunk.append(eos_id)
np.array(chunk, dtype=np.uint16).tofile(f)
We store IDs as uint16, two bytes per token. A vocabulary of 8,192 fits easily, since 16 bits hold values up to 65,535, and the file ends up half the size of 32 bit storage. Later, np.memmap lets us read slices of the file without loading all of it into memory.
The results: 2,119,719 training stories became 467,094,292 tokens, a 934 MB file, in about eleven minutes on a free CPU notebook. The 21,990 validation stories became 4,694,730 tokens.
Two mistakes worth admitting
We checked the files by reopening them and asserting that the number of end markers matched the number of stories. That check failed on our first full run, and it was right to. The files were fine. Our token counter had not been reset between the validation and training loops, so it carried over the validation total and reported 471,789,022 instead of 467,094,292. The difference was exactly the validation token count. Without the assertion we would have trusted a wrong number.
The second mistake was smaller. We uploaded our data without saving the tokenizer next to it. Token IDs mean nothing without the tokenizer that produced them, so always keep the tokenizer file in the same folder as the data.
What comes next
We now have a trained tokenizer, a verified dataset of nearly half a billion tokens, and a clear understanding of what is inside both. The next stage is the model itself: turning token IDs into vectors, building attention from scratch, and training a small Transformer to predict the next token.
Frequently asked questions
What is BPE in simple terms?
Byte pair encoding is a method that starts with single bytes and repeatedly merges the most frequent neighbouring pair into a new token. Common words end up as one token, and rare words are split into pieces.
Why do language models use byte level BPE?
Because it can encode any text in any language without unknown tokens. The base vocabulary is just the 256 possible byte values, and merges make common sequences short. GPT-2 popularized this approach, and many modern models use the same idea.
How big should my tokenizer vocabulary be?
It depends on your data and model size. Small models trained on simple text work well with a few thousand tokens. General web text usually needs tens of thousands. Larger vocabularies shorten sequences but increase the size of the embedding table.
Why not just use words as tokens?
Word vocabularies are huge, and any unseen word becomes an unknown token. Subword tokens avoid both problems.
Why store token IDs as uint16?
If the vocabulary has fewer than 65,536 tokens, two bytes per ID is enough. It halves disk and memory use compared with 32 bit integers.
Leave a Reply