What is multimodal AI? One brain that listens, talks and sees

Where do the senses live?

Last time, AI made a video from one sentence. Listen, talk, watch, make. So where do those senses live inside AI?

THE OLD WAY

Many separate machines glued together

The old way glued many machines together: one turns voice into text, one describes pictures in text, then everything goes to the machine that reads text. With each handoff, information gets lost, like a game of telephone.

SMALL PIECES

Everything becomes small pieces

The new way: everything is cut into small pieces called tokens, like in the video about tokens. Text becomes pieces, pictures become little squares, sound becomes short clips. It all goes into one brain.

ALL AT ONCE

It gets both tone and picture

Hearing sound directly, it can pick up tone: whether "fine, whatever" is said happily or sulkily. Seeing a picture directly, it can read charts and see where you're pointing.

NOTE

Many senses, still can be wrong

But many senses doesn't mean no mistakes. For important things, you're still the one who checks.

RECAP

Three series in one picture

Looking back: AI learns by turning billions of knobs itself. It's graded by tests, arenas and your own test. And now it listens, talks, watches and makes, all in one brain.

THANK YOU

Thanks for coming along

Thank you for coming along for all 18 parts. Just got here? Go back to part 1, "How AI learns".