Opening up AI's brain
You use AI every day, but what is it really doing inside? This eight-part series opens up AI's brain. Part one: AI doesn't read words the way you do.
Scissors that cut sentences
Before it understands anything, AI cuts your sentence into small pieces called tokens. Some pieces are a whole word, some are just part of a word.
Each piece becomes a number
Then each piece is turned into a number. What AI actually sees is a list of numbers, not letters.
Why AI miscounts letters
That's why AI often miscounts letters. It doesn't look at each letter, it looks at whole pieces. Letters hidden inside a piece are easy to miss.
Some languages take more pieces
Languages with accents, like Vietnamese, often get cut into smaller bits than English. The same idea takes more pieces. And how much AI can read, and how much you pay, are both counted in pieces.
So how do numbers carry meaning?
But if each piece is just a number, how does AI know "cat" is closer to "dog" than to "truck"? Part two has the answer.
This article is based on the video AI doesn't read words, it reads "pieces" from the Mark học AI channel. Watch the video (in Vietnamese) to see the animations.