[CVPR'21 WAD] Keynote - Andrej Karpathy, Tesla

This talk was worth watching (any Karpathy or Carmack talk is imo): ▶ this video Specifically this is the relevant part: ▶ this video They removed radar because it was making things worse.

Radar has poor resolution, which means it can't always differentiate between an object near the car's path and an object in the car's path. The radar can give false positives when there's a truck on the side of the road or a metal sign on an overpass. Then you have to use the vision cameras to figure out what's really going on. Radar also loses its "lock" sometimes, which can be an issue with cars braking quickly. The radar will say "no obstacles detected" when the car in front of you is slamming on the brakes. Again, you have to use the vision cameras to figure out what's actually going on. The people at Tesla decided that the radar was creating more noise than it was helping with, so they removed radar last year. Andrej Karpathy explains it in far more detail in this talk: ▶ this video

> These are the same things that can cause a traffic accident for a human with human eyes, so there's really no logical reason to assume that you can create a more performing and safer computer driver by emulating human vision only. Okay. This is a concrete claim that I think is falsifiable. I'll give not just one, but multiple arguments. Argument 1: Connect Four, Chess, Checkers, Go, Limit Hold'em, No Limit Hold'em, Poker, DOTA, and Starcraft II are all examples of games in which AI outperforms even the best humans despite having equal sensory access. Driving can be reduced to a problem comparable to playing an imperfect information game and we can play imperfect information games at superhuman levels using computers. Therefore, it is reasonable to expect that even human-level sensory access can enable safer than human performance. Argument 2: Currently we have statistics on driver-assistance programs that demonstrate that human + technology assistance is superior to human alone. This holds true even in cases in which those assistance technologies are only using vision. So we have a contradiction. If it wasn't possible to do better than human with only human senses, we wouldn't be able to point to many different situations in which driving with an AI clearly outperforms humans, but we can. For at least those subsets of AI driving, we can say that an AI driver will outperform humans. > I can't understand why they removed radar. Check out ▶ this video for a very detailed breakdown

If you're interested, Tesla's head of AI recently did a public presentation of their new vision based depth system. It's worth a watch if you find this stuff interesting and enjoy learning about the forefront of technology. ▶ this video A key takeaway is that they've been running this stack in shadow mode (validating output but not controlling the car) on everyone's Tesla for quite some time. Equivalent to 1000 YEARS worth of real world driving. And from this data they've proven it is now superior to radar in all circumstances.

He is talking about the long technical talk at CVPR. ▶ this video For example look at this section: ▶ this video Also if you look into it you will find Google's self-driving car project ran into similar issues. They upgraded the radar in response to the issues rather than removing it, but there are actual issues. Vision only was better than radar plus vision in their stack. For real world situations that frequently come up in driving radar wasn't worth it. The theoretical accuracy of the sensor fusion didn't happen in practice because in practice you needed to have vision tell you when sensor fusion wasn't going to be accurate. It isn't naive sensor fusion as in toy math models of sensor fusion. So when there was a disagreement you needed to go with vision. Yet vision was already able to predict both the failure of and the actual result of radar. In theory, you could just have done the more complicated sensor fusion. In practice, the relative advantage of improving other parts of the vision stack far exceeded improving the sensor fusion. Over time one would expect the relative advantage to change as other improvements were made such that this was no longer true. That doesn't mean the time at the decision wasn't correct, it just means people who evaluate it with future changes already in place are suffering from hindsight bias/committing an anachronistic fallacy.

Karpathy covered it during a conference keynote here: ▶ this video It also is pretty common with other vehicles, they just don't get as much press as Tesla: link link

Appreciate the downvote/disagreement. But here is Karpathy explaining how vision can be used to measure distance to objects accurately[0] Here is the fact that LiDAR doesn't work in the rain[1]: "... In heavy rain, for example, the light pulses emitted from the lidar system are partially reflected off of rain droplets which adds noise to the data, called 'echoes'." Which logically implies you need to revert to vision, as see [0] also for why Radar is unreliable. [0] ▶ this video [1] link

> why not use binocular forward facing vision? Because Tesla have demonstrated that it's unnecessary. The depth information they are getting from the forward-facing camera is exceptional. Their vision stack now produces depth information that is dramatically superior to that from a forward-facing radar. ▶ this video (It's also worth noting that depth information can be validated when the vehicle is in motion, because a camera in motion has the ability to see the scene from multiple angles, just like a binocular configuration. This is how Tesla trains the neural networks to determine depth from the camera data.)

From the keynote of CVPR. CVPR is the premier annual computer vision event comprising the main conference and several co-located workshops and short courses. link ▶ this video

No. Newer versions of Tesla's software disable the radar on cars that shipped with it.[1] Vision has gotten so much better (thanks in part to training from radar data) that radar adds noise to their model of the outside world. Andrej Karpathy explained the issues in a talk this summer.[2] 1. link 2. ▶ this video

Given that radar is functionally blind[1] to static objects when travelling at high speeds, I'd argue it's not just a good thing, it's a necessary thing. Tesla have demonstrated the superiority of their vision approach and validated it with 1,000 years of real-world driving, running it in shadow mode on customer vehicles.[2] [1] Technically radar isn't blind to static objects, but since it cannot disambiguate them from tunnel entrances and highway overpasses, the data is useless. Therefore it's functionally blind. [2] ▶ this video

IIRC, in this[1] video Andrej Karpathy mentions radar was mostly redundant, and in addition was occasionally harmful (sending conflicting signals vs. video), so they got rid of it to reduce cost and complexity. [1]: ▶ this video