Meaning depends on nearby words
Last time, each word had a spot on the map of meaning. But a word's meaning also depends on the words around it.
The "it" puzzle
Read these two sentences. "The cat didn't jump over the wall because it was too tired." And "because it was too high". You know right away that "it" is the cat in one and the wall in the other. How does AI know?
Threads of attention
The answer is the attention mechanism. Each word "looks" at every other word in the sentence and gives a score: which word matters most to me? Tired goes with cat, high goes with wall.
The whole sentence looks at itself
Every word does this at the same time, repeated across dozens of layers. That's the heart of every chatbot today.
Important lines get buried in the middle
But attention has limits too. When you send a long text, an important line stuck in the middle is easier to miss than one at the start or the end.
Who teaches this brain?
The machine is built. But who teaches it so much? Part four has the answer.
This article is based on the video Attention: where does AI focus? from the Mark học AI channel. Watch the video (in Vietnamese) to see the animations.