Andrej Karpathy - AI for Full-Self Driving at Tesla

You can learn about their systems by watching talks by Andrej Karpathy. As a robotics engineer interested in vision, their architecture is inspiring. This talk [1] is a good overview but each talk he gives is a little different so search for more if you want to know as much as possible. But the big thing is that their autonomy computer can be programmed to look for odd scenarios and send them back home. Tesla uses their fleet of hundreds of thousands of cars to collect edge cases like this, and then they have a kind of compartmentalized neural network system that breaks apart disparate tasks. With their collected examples they can create unit tests to ensure that the moon stops activating the stoplight detector. Once trained, the unit tests presumably help ensure they don't end up with future regressions. So basically every time you see a Tesla do a weird thing, there is a good chance it will stop doing it soon enough. At least if it's hitting hacker news. [1] ▶ this video

Removing the cameras may be a good idea, but note that their extensive fleet of camera-equipped vehicles is an important part of their potentially VERY lucrative self driving software development. EDIT: I highly recommend watching head of AI at Tesla Andrej Karpathy's talk on their self driving architecture. "The Fleet" as a source of data is a vital functional part of their self driving software development. They may be able to calculate that including cameras and a computer on a car that will ship in high volumes would increase sales of their self driving option across all Tesla vehicles (including that low cost model) enough to justify the inclusion of those parts. ▶ this video

If you actually believe that Tesla will fail to deliver full self-driving in the coming years, I have two questions. 1. have you watched this entire technical presentation made by Andrej Karpathy, Senior Director of AI at Tesla? ▶ this video 2. if you understand what you've seen in that video, why do you think Tesla will fail?

This is anti-Tesla astroturfing, so I flagged it. If you'd like a high-level technical overview of Tesla's FSD efforts, check out this talk by Andrej Karpathy from February 2020: ▶ this video There are literally thousands of videos of people filming the FSD beta doing amazing things. They are making great progress very, very quickly: link As you can see from videos like the one above, real-world data is needed to make FSD practical. There is a very long tail of things to get right, so blocking this progress with regulations would effectively outlaw FSD, or delay its development by a very long time. Given the number of road accidents that human drivers cause (think of just sleep deprivation and drunk driving), delaying FSD through regulation would ultimately cause a great number of uncessary deaths.

It's not the same: ▶ this video You can watch that talk and see the approach they're taking. Maybe you're more skeptical than they are about the near term possibility, but you can see that the work and progress is real. The AGI risk is real too. What Milton said doesn't make semantic sense, it's not a question of timelines. link

Better yet, let's get rid of cars altogether from city streets. This is coming along very nicely... link Autonomous driving is just around the corner, too, which means that there is no need to waste space on parking structures, either. Here's a Tesla engineer that gave a great technical talk demonstrating their progress as of early this year: ▶ this video EDIT: yes, there are clear reasons why having an three-dimensional underground structure will absolutely affect the social dynamics above-ground in a city, yet it remains true that it is possible to banish cars from downtown cores in the near future (i.e., say by 2030 or so), given the present shift to electrification and autonomous driving. These issues are worthy of debate, and I'm just trying to shift the conversation to what's current. I'm not trying to rain on anyone's parade! I love the goal of city streets for people, as exemplified by the beautiful laneways in Melbourne (11 minutes): link

Here's the most recent talk by Karpathy about this topic: ▶ this video (there are more if you google for "karpathy talk") Here's the summary via 2 examples. 1. Example of gathering data that needs further labeling To implement neural network (NN) to recognize stop signs they program the cars to recognize things that look like stop signs and deploy that to the fleet of over 600 thousand cars on the road. The cars send those "might or might not be a stop sign" images back to tesla and they get manually labelled and added to a training set. They gather enough images and the "recognize stop signs" feature is done. Apply the same logic to other recognition tasks: recognize cars, pedestrians, animals, speed signs, traffic lights etc. Yes, labeling is expensive but everyone else (including Waymo) has the same cost. But Waymo has even bigger cost of driving around to collect those images. Tesla makes gross profit on every car they sell and they got 600 thousand people driving for them for free. Waymo has 600 cars, each of them reportedly costing over $200k and they have to pay drivers at least minimum wage for each hour of driving. That's why Google is reportedly is spending $1 billion a year on Waymo and why they need $3 billion of additional investment to keep going. 2. Example of gathering data that doesn't need labeling Consider implementing NN to recognize cut in i.e. other cars entering your lane in front of you. They deploy a first version trained on small sample, running in shadow

They are reliable enough to use for FSD: ▶ this video so i think it's reliable enough for park assist, summon etc. Yeah you're def right about the moisture sealing... it's been hit or miss on production tolerances

The actual report is here: link It failed to detect a stationary mannequin in one specific scenario. Does this generalize? Does this mean there's a major safety problem? These would be assumptions that this test can not prove. A comprehensive safety analysis has to also include these scenarios: ▶ this video

> All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update. What is this evidence? I've seen a few talks from Andrej Karpathy that indicate to me a more deliberate approach.[0] "Software 2.0" itself seems like an approach meant to systematize the development, validation & testing of AI systems, hardly a seat-of-your-pants approach to releases. I have my own criticisms of their approach, but it seems there is pretty deliberate care taken when developing models. [0] ▶ this video

There is a mix of a "neural net planner" and a "explicit planning and control" in traditional code[1]. The explicit planner has the last word, and uses input from both the "vector space" and the neural net planner. Karpathy has commented about the neural nets gradually replacing the traditional code. See his presentation[2] around 18:45. [1] link [2] ▶ this video

Not yet. I have a four wheel drive robot I designed with four 4k cameras feeding in to an Nvidia Jetson Xavier. [1] Just getting it to navigate itself using vision would mean building a complex system with a lot of pieces (beyond the most basic demo anyway). You need separate neural nets doing all kinds of different tasks and you need a massive training system for it all. You can see how much work Tesla has had to do to get a robot to safely drive on public roads. [2] From where I am sitting now, I think we are making good inroads on something like an "Imagenet moment" for robots. (Well, I should note that I am a robotics engineer but I mostly work on driver level software and hardware, not AI. Though I follow the research from the outside.) It seems like a combination of transformers plus scale plus cross domain reasoning like CLIP [3] could begin to build a system that could mimic humans. I guess as good as transformers are we still haven't solved how to get them to learn for themselves, and that's probably a hard requirement for really being useful in the real world. Good work in RL happening there though. Gosh, yeah, this is gonna take decades lol. Maybe we will have a spark that unites all this in one efficient system. Improving transformer efficiency and achieving big jumps in scale are a combo that will probably get interesting stuff solved. All the groundwork is a real slog. [1] link [2] ▶ this video [3] link