What is a token in AI?
Also called: tokens, tokenization
A token is a small piece of text that a language model reads and writes, often part of a word. Models process text as tokens, charge by the number of tokens, and have a limit on how many tokens they can handle at once, called the context window.
How does a token work?
Text is split into tokens before the model sees it. Common English words are often one token; rare words, numbers and many Indian-language words get split into several. A rough rule: 100 tokens is about 75 English words.
Why does a token matter?
Tokens decide how much an AI request costs and how much text fits in one conversation.
An example
Example: 'Namaste' might be split into two or three tokens, while 'hello' is usually one.
Related terms
Learn it properly
A token is covered in AI Builder, with short lessons, practice questions and instant feedback.