In plain words

token

A basic unit of text, such as a word or part of a word, that a language model processes.

A bit more detail

Language models break sentences into tokens to compute meaning and generate answers. Cloud providers often charge customers based on how many tokens they feed into and receive back from an AI model.

Stories that use this word