Skip to content

Glossary:

Tokenization

Tokenization is the process of breaking down text into smaller units, called tokens. These tokens can be words, subwords, or even individual characters.

Find out more
Back to glossary

Share this