Tesla AI Day 2021

I spent time working with Andrej and the rest of the FSD team back in 2020/2021, and we had plenty of conversations on how human visual processing maps onto our neural network architectures. Our approach—transformer-based attention blocks, multi-scale feature extraction, and temporal fusion—mirrors elements of the biological visual cortex (retina → LGN → V1 → V2 → V4 → IT) which break down raw inputs and integrate them over time. It’s amazing how closely this synthetic perceptual pipeline parallels the way our own brains interpret the world. The key insight we discovered was that explicitly enforcing brain-like topographic organization (as some academic work attempts - such as this one here) isn't necessary - what matters is having the right functional components that parallel biological visual processing. Our experience showed that the key elements of biological visual processing - like hierarchical feature extraction and temporal integration - emerge naturally when you build architectures that have to solve real visual tasks. The brain's organization serves its function, not the other way around. This was validated by the real-world performance of our synthetic visual cortex in the Tesla FSD stack. Link to the 2021 Tesla AI day talk: ▶ this video

I believe Tesla uses synthetic data for FSD/Autopilot training as well. Not sure what's the ratio between synthetic data and real videos, but it seems that they plan to rely more onto the first group in the future. ▶ this video

Machine Learning and Neural Networks are not applicable for solving all kinds of problems. In fact, in Tesla AI Day Elon Musk remarked that "I discourage the use of machine learning... unless you have to use machine learning, don't do it" [1]. Getting machine learning to work right is hard. If it can be avoided it should be. So to cast this as Software 1.0 vs 2.0 doesn't make sense. There is a class of problems where neural networks work better. Everywhere else we will continue to use traditional code. [1] ▶ this video

The Tesla AI day videos [1] go into some detail about this. They use multiple networks that are dedicated to specific tasks. [1] ▶ this video (2021), link (2022)

Waymo has superior performance based on their historical statistics. It makes sense, since their lidar sensors capture more of the environment, and directly in 3D. Their AI also seems better QA’ed. The Tesla AI Day[0] surprised me as it showed they only had a simple architecture for a very long time, simply feeding barely processed camera pixels to a DNN and hoping for the best with little more than supervised learning off human feeds. Their big claim to glory was that they rearchitectured it to produce a 2D map of the environment… which I thought they had years ago, and is still a far cry from the 3D modeling that is needed. After all, sure, we humans only take two video feeds as input… But we can appreciate from it the position, intent, and trajectory of a wealth of elements around us, with reasonable probability estimates for multiple possibilities, sometimes pertaining to things that are invisible, such as kids crossing from nowhere when near a school. Cruise also seems to have better tech; they had a barely-watched 2h30 description of their systems[1] which shows they do create a richer environment, evaluate the routing of many objects, and train their systems on a very realistic simulation, not just supervised training, which means it can learn from very low-probability events. They have a whole segment on including the probability that unseen cars may travel from perpendicular roads; Tesla’s creeping hit-or-miss are well-documented on Youtube. [0]: ▶ this video [1]: link

link This is a rather intuitive mathematical proof that neural nets can emulate all functions in many dimensions. Knowing integrals (Calc 102) would help. With this information it’s either that self-driving is eventually possible or human brains are not mathematically representable (i.e. something akin to a soul). Meanwhile, the data collection/storage is getting better (all Teslas are continually testing FSD and reporting failure cases, synthetic generation of “good and unique” perfectly labeled data[0] is only a year or two “old”). Sensors/cameras are improving (and have a lot more room to improve). The compute hardware is getting better and cheaper (analog computers[1], TSMC 2nm and beyond, Apple/Google/Teslas neural cores, etc). Additionally, given Waymo is already in production without humans in PHX it’s arguable that we already have “autonomous vehicles”. We now just need them to improve slightly and for Waymo or similar to test in the top 25 cities in the US. Which is mostly a matter of capital and time not technology limits. Maybe weather will be an issue for a few more years/decades - but even if Waymo only operated in Summer, I’d be happy to reduce my Lyft costs - and with enough data and maybe better sensors the neural nets will be able to handle weather too. Especially if the car just goes slow. [0] ▶ this video [1] link [2] link

Cruise actually consider both social dynamics and uncertainty (i.e. what can hide behind an obstacle, or where are pedestrians/bikes/cars likely to move to). If you are interested in self-driving cars, I can highly recommend their presentation from November 2021: link For me it felt more convincing than Tesla's (a few months prior); ▶ this video

> The "images" are part information. This. It is a mashup of both. I can clearly visualize many 3D objects in my mind, as well as zoom, pan, scale, rotate, recolor, animate, distort them, etc. But it is not purely visual, because I often "see" opposite side of a cube momentarily, which I should not be able to from this perspective. So, it is not a rendering engine. It is not physically, spatially and temporally stable. Stability is lost especially as your focus shifts (e.g. "seeing" behind objects as your attention slips there) The best way I know to describe this is by old and new Tesla FSD predictions which are sampled here: ▶ this video Improved version is how people talk about visualization, but the old unstable version is how (at least for me) it mostly is.

This is a poor use case for ML. You'd only have a couple hundred examples for a neighborhood with maybe two dozen variables. The model will also miss key variables that that there aren't datasets for but would be apparent upon a visit. ML only beats humans in very, very narrow scenarios. ▶ this video

Here's Tesla's AI day presentation on how they are going to use AI in their planning system because traditional methods aren't scalable enough: ▶ this video (1:20:15). You can see hints of this in presentations from the other self-driving companies. Boston Dynamics was definitely an exception for a long time. Certainly the perception systems that they are adding use ML, but you are right that they use expressly use classical methods for their control system.

I mean in the same event that you list, Elon literally opens with: "Tesla is much more than an automotive company, and we have lots of deep AI activity..." I think it is fair to say that a company that is literally inventing its own AI training supercomputer is probably an AI company as well :) ▶ this video

I don't know if anyone else caught this, but their "optimal trajectory" planned path is illegally changing lanes in an intersection, at -1:26:00: ▶ this video . Pretty funny.