Tokenizer
TechnologiesThe component that splits text into tokens, the units the model actually reads. A multilingual tokenizer encodes Italian in fewer tokens than an English-only one, reducing compute and context consumption.
The component that splits text into tokens, the units the model actually reads. A multilingual tokenizer encodes Italian in fewer tokens than an English-only one, reducing compute and context consumption.