Sensor fusion: how AI combines LiDAR, radar and camera

A rainy night, dark clothes

On a rainy night, a person in dark clothes steps into the road. The camera is only half sure it's a person, radar sees something moving slowly, and LiDAR sees a shape 1.7 meters tall.

HOW IT WORKS

Three clues that match

Combining several sensors into one conclusion is called sensor fusion. The human shape, the walking speed and the blurry figure on camera all match, so the AI is sure enough to brake.

STEP 1

Each eye reports a number

For distance, each eye reports its own number, and each has its own error. The camera's error is the largest.

STEP 2

Trust some eyes more

The AI doesn't take a simple average. It trusts the eyes with smaller errors more. Worked out, LiDAR gets 58 percent of the weight, radar 37 and the camera 4, giving 23.8 meters.

RESULT

Combined is more accurate

The channel ran 10,000 trials with a true distance of 24 meters. LiDAR was off by 64 cm on average, while the combined estimate was off by only 49, better than even the best single eye.

COMMON MISTAKES

When the eyes disagree

Fusion makes mistakes too. When the eyes report opposite things or the AI misreads them, the car can brake hard on an empty road, known as phantom braking. In 2022 the US opened an investigation after hundreds of complaints like this.