What is gradient descent? How AI reduces error, like walking down a foggy mountain

Which way to turn?

Last time: AI measures how wrong it is with a number called the loss. Now the machine needs to know: which way should it turn the knobs to make that number go down?

EXAMPLE

Blindfolded on a mountain

Picture yourself blindfolded on a foggy mountainside. You want to get down to the valley, but you can't see anything.

FEEL THE SLOPE

Feel with your feet

The only way: feel around with your feet to find where it slopes down the most. Then take a small step that way. Feel again, step again.

THE ERROR MOUNTAIN

Height is the error

For AI, height is the error. Each spot is one way of setting the knobs. Going down means being less wrong. This way of learning is called gradient descent.

REPEAT

Millions of small steps

The machine repeats feeling and stepping many, many times, sometimes millions. Each step turns all the knobs at once, and the error drops a little more.

CAREFUL

Long steps or short steps?

Step size matters a lot. Steps too long, and you jump right over the valley and bounce up and down. Steps too short, and you walk forever without getting there.

PART 4

Which knob caused the mistake?

But there are billions of knobs, and the mistake only shows up at the very end. How do you know which knob is to blame? See you in part 4.