Vietnamese uses more tokens
The same idea written in Vietnamese usually uses more tokens than in English.
TOKEN
AI reads in pieces
AI doesn't read word by word but piece by piece, and each piece is called a token, as video 19 of the channel explained.
TONE MARKS
Words with tone marks get chopped up
Models learn far more English, so Vietnamese words with tone marks often get split into more, smaller pieces.
THE EFFECT
Costs more, remembers less
More tokens means using the API costs more, and long conversations fill the context window faster.
This article is based on the video Why Vietnamese uses more tokens from the Mark học AI channel. Watch the video (in Vietnamese) to see the animations.