Token
T
Token
Definition
The basic unit of text that language models process. A token can be a word, part of a word (subword), or a single character, depending on the tokenizer. Token counts determine model input limits, output length, and API pricing.