You’ve probably wondered what is natural language processing. It is the branch of AI that teaches computers to work with human language by turning text into numbers they can calculate with.
If you want the bigger picture first, start with what is artificial intelligence, then come back here for the language side.
| Key fact | Value |
|---|---|
| GPT-2 vocabulary size | Roughly 50,000 tokens |
| Toy example “the cat sat” after tokenization | 3 token IDs (0, 1, 2) |
| “Tokenization” split by a sub-word tokenizer | 2 pieces (token + ization) |
| Example document length | 5,000 tokens |
| Example model context length | 4,096 tokens |
These figures come from a DEV Community tokenization explainer, and they are illustrations rather than universal constants. Models and tokenizers change, so check the source before you rely on them.
Why computers cannot read words directly
To a computer, a sentence is just a string of characters. A neural network is, at its core, a system of matrix multiplications and other numerical operations, so it cannot multiply a matrix by the letters c-a-t.
That is why every NLP system, whether it is rule-based, statistical, RNN / LSTM or transformer, needs a conversion step: raw text in, numbers out. Natural language processing is the work of building that bridge and then doing something useful on the far side, such as translating, summarizing or judging the sentiment of text.
How tokenization turns text into numbers
Tokenization is the process of splitting text into discrete units, called tokens, and assigning each one a unique numerical ID. It is the literal first step of the pipeline.
A token can take several forms:
- A whole word, such as “cat”.
- A sub-word piece, such as “token” or “ization”.
- A single character.
- A punctuation mark, which gets its own token.
Before IDs can be assigned, the system needs a vocabulary: a fixed list of every token the model knows, each mapped to an integer. With a tiny vocabulary that holds “the”, “cat”, “sat”, “on” and “mat”, the sentence “the cat sat” becomes the ID sequence 0, 1, 2.
Real vocabularies are far larger. The vocabulary of GPT-2, an early large language model, has roughly 50,000 tokens, and it is fixed at training time.
What happens with words a model has never seen?
Early tokenizers split on whitespace and punctuation, so each word became one token. The downside was that they could not handle words missing from the vocabulary, and such a word was replaced by a generic unknown token, which threw away its information.
Modern systems use sub-word tokenization instead. A rare or unseen word gets split into pieces the vocabulary does recognize, down to single characters if needed, so almost any input can be represented.
Byte-pair encoding, step by step
Byte-pair encoding (BPE) is the most widely used sub-word algorithm, and it is also how the vocabulary gets built. It runs like this:
- Start with individual characters as the base vocabulary.
- Find the most frequent adjacent pair of tokens across the training text.
- Merge that pair into a new token.
- Repeat until you reach the target vocabulary size.
In the explainer’s example, a corpus dominated by “lowest” and “newest” first merges e and s into es. The next round then merges es and t into est, so common endings become single tokens.
Languages without spaces
BPE assumes the text is already split into words by whitespace. That works for English but breaks down for languages that do not use whitespace to separate words, such as Japanese and Chinese.
SentencePiece solves this by treating the input as a raw stream of characters, including the whitespace itself. This makes it language-agnostic, which is why it is widely used in multilingual models.
The classic natural language processing toolkit beyond tokenization
Tokenization is only one step in a longer chain. A survey titled Deep Learning and Machine Learning – Natural Language Processing: From Theory to Application, in a version dated 18 Nov 2025, devotes a whole chapter to text preprocessing.
Among the techniques it lists are:
- Stopword removal, and lemmatization and stemming.
- TF-IDF and bag of words.
- Part-of-speech tagging and named entity recognition.
- Word embeddings such as Word2Vec and GloVe.
- Topic modeling with LDA.
- Contextual word embeddings such as BERT.
The same survey also covers language detection and translation, text summarization preprocessing and sentiment analysis preprocessing. A later chapter on data cleaning for training large language models shows that preparing text still matters even with modern neural models.
Context limits and where NLP fits in AI
Every transformer model has a maximum number of tokens it can process in one input, called its context length or context window. If a document tokenizes to 5,000 tokens and the model’s context length is 4,096, the input has to be truncated or split.
In practice that means either cutting off whatever does not fit or splitting the document into smaller chunks that are processed separately. The chunking approach is relevant to Retrieval-Augmented Generation, where large documents are broken into meaningful pieces before being fed to a model.
Language models also have to learn from data, and the choice between labelled and unlabelled examples is covered in supervised vs unsupervised learning. Text is not the only kind of input either, since the survey also lists handling multimodal data such as text with images or audio. For the image side of that story, read what is computer vision.
Frequently asked questions
What is natural language processing in simple terms?
Show answer
It is the field that lets computers work with human language. Because a neural network only operates on numbers, the text is first converted into numerical IDs, and models then work on those numbers.
Why do models use sub-word tokens instead of whole words?
Show answer
A whole-word vocabulary cannot handle words it has not seen, such as rare terms, typos or names. Sub-word tokens let the model split an unseen word into smaller known pieces instead of discarding it.
What is a context window?
Show answer
It is the maximum number of tokens a transformer model can process in a single input. A document longer than that has to be truncated or split into chunks.
Why is SentencePiece used for multilingual models?
Show answer
It treats text as a raw stream of characters rather than assuming words are separated by spaces. That makes it language-agnostic, including for Japanese and Chinese.
