Cars and robots

I’ve got two new bets relating to AGI timelines, this time with Nick Whitaker and roon. One is about cars, and the other about robots.

Like with my previous bet with Daniel Kokotajlo, my intention is to try get short time-line, hard-takeoff types to be concrete about their expectations. If there happens to be deep flaws in the assumptions of this group, I think having these concrete examples will make it easier to diagnose what those mistaken assumptions were, after the fact.

I say this as someone who till recently was onboard with the short time-line worldview. In 2017, I expected AGI by 2022, followed by an immediate hard take-off. In 2020, AGI had been pushed back to 2030 in my mind (but still hard take-off within a couple years). Today, I sit somewhere around 2035 for “AGI proper”, and a slower takeoff for ASI (who knows, maybe more than 15 years later). This change in position occurred over four years studying philosophy at university, where I had time to unpack the intuitions I had gained studying deep learning in 2017-2019.

For context, the “short time-line, hard-take off” position I refer to is that broadly found in the four recent “ASI manifestos”: Situational Awareness: The Decade Ahead by Leopold Aschenbrenner, The Intelligence Age by Sam Altman, Machines of Loving Grace by Dario Amodei, and AI 2027 by Daniel Kokotajlo. These were all published within the space of a year (June 2024 - April 2025); Sam and Dario are the CEOs of OpenAI and Anthropic, and the other two authors previously worked at OpenAI.

It’s this particular nexus of worldviews that I am targeting — trying to show that, while these individuals differ in major ways, there are some underlying, shared assumptions between them that might be off.

The bets

Bet 1:

By the beginning of 2030, there will be fully autonomous cars capable of driving on most city roads with lower incidence of critical accidents than the average human driver. “Fully autonomous” means no human intervention (including teleoperation).

Bet 2:

By the beginning of 2040, there will be fully autonomous robots that can replace chefs and waiters in most restaurants. “Fully autonomous” means no human intervention in the activities performed in the usual duties of chefs and waiters in restaurants today. Furthermore, the restaurants must be largely unmodified to assist the robots. Robots must be capable of stepping into the role of human waiters and chefs in the typical restaurant of today (even if the unit economics do not make sense).

Nick is saying yes, I’m saying no, at 1:1 odds (50% implied odds). Nick’s put up $1 for each — not crazy stakes, but to be fair he’s COO of a billion dollar hedge fund focusing on AI progress.

50% represents to me something like the fair odds. I’ve asked Nick, and he’s said his fair odds are 90% (cars) and 80% (robots).

Rationale

I’ve always felt that FSD (full self-driving) was right around the corner, and then I’ve always been surprised when it wasn’t. Apparently this happens to the best of us.

I think it’s especially tempting at the moment to feel that FSD is right around the corner. Right now, almost everyone (at least in the US) can go outside and experience it. In San Fransisco, Waymo is conducting hundreds of trips a day with zero driver in the car. And people with FSD 14 are experiencing end-to-end commutes with zero human intervention.

You don’t need to believe in AGI or ASI by the end of the decade. The felt experience of the matter is that FSD is practically here. Given we essentially only had lane assist at the start of the decade, it seems reasonable that scalable, human-level self-driving is a lock by the end of the decade.

But this is all felt sense. Let’s have a look at the data.

This is from George Hotz’s presentation at COMMA CON 2025. FSD 14.1+ is clearly a step-wise shift in the capabilities of FSD. Precisely, it’s a step-wise shift to ~3,000 city driving miles before critical disengagement.

Humans crash approximately every 500,000 miles. And so George argues that this improvement, while large, is still 2 OOMs off human level driving. If we take the trend of improvement over the history of work on FSD, these two OOMs (orders of magnitude) will take approximately 8 more years of work. (Or we could take the trend over the past 6 months, in which case we’ll get FSD tomorrow).

I maybe believe a little more in exponentials than George does, and so I’m happy to say that this 14.1+ step-wise shift is a step-wise acceleration, and so maybe it’ll take 4 years to get accidents below human level in fully autonomous systems. And I’m happy to be wrong 50% of the time, if it turns out that AI R&D acceleration occurs, among other things.

(”But what about Waymo?” George goes into deeper reasoning behind his eight year estimate in the presentation, and there’s good reason to take him seriously, given his track record of making well-grounded predictions around engineering challenges in this field).

Why the felt sense is off: reasoning about tiny numbers

“But my Tesla just drove me to work, with zero interventions. Full-self driving is already here!”.

The supermajority of adults have never been in a car accident that involves a fatality or serious injury. This makes sense, given accidents occur approximately every 500,000 miles (and fatal accidents kill you half the time!). Minor accidents occur approximately 3-4 times in a human lifetime.

So this is the first problem with the intuition that cars will be safer than human drivers very soon. A fully autonomous car could be half as safe as the average human driver, and yet you’d have to be driving in it for ~10+ years to experience this degraded safety. (There’s a funny world where human driver safety gets to 1,000,000+ miles, because half the cars on the road are autonomous and easier to navigate around as a human driver, autonomy means fatigue related accidents are lowered, and yet autonomous cars still crash every 100,000 miles because they lack one or two minor things about high-order human perceptual awareness, i.e. they get wigged out on 1 in 100,000 mile edge cases).

The second problem is the difficulty with feeling small numbers. If you go from 10% to 99% drives without disengagement, you get an order of magnitude increase in reliability; the felt sense of this change seems incredible. Yet going from 99% to 99.9% drives without disengagement is also an order of magnitude increase in reliability, but you’re not going to notice it. The real “experience” of the improvement would take more than a month, rather than within the same day for the first OOM. But from an engineering perspective, we may be talking about the same rate of engineering improvement to get from one OOM to the other. Indeed, we may be talking about a slower rate, if the nature of the engineering problem has gotten more difficult in various ways.

Of course, the engineering problem may have gotten easier. Maybe they have figured out something with 14.1+ at a training algo level that means they are now at the bottom of a really powerful sigmoid, and Tesla will cover those two OOMs in a couple years, or a couple months. This is why I’m happy to call it a coin flip.

Reasons we might not be one sigmoid away

How many sigmoids are left (before AGI), before we hit the exponential (ASI)? That’s the question.

Having followed deep learning for 10 years, there seems to be a consistent pattern of underrating the number of sigmoids, roughly proportional to how close you are to the action. It’s this mental phenomenon that I want to get at:

“2023 was the moment for me where AGI went from being this theoretical, abstract thing. I see it, I feel it, and I see the path. I see where it’s going. I can see the cluster it’s trained on, the rough combination of algorithms, the people, how it’s happening. Most of the world is not there yet. Most of the people who feel it are right here [in San Fransisco]. A lot more of the world is going to start feeling it. That’s going to start being intense.” — Leopold Aschenbrenner (23:00)

A small number of people had this moment in 2011. Some more had this moment in 2014. Then again in 2016 (Move 37). Then again around 2020, and so on. Indeed, each time it happens you tend to get a new frontier lab!

I’m not saying we definitely won’t get “proper AGI” (say where the AGI is doing better AGI R&D than a human) before the end of the decade. At some point, the “right around the corner prediction” will be spot on. But there seems to be a consistent misapprehension of the number of sigmoids lying in the way before the exponential.

Per the previous section, what does that seem to be? It looks something like “from where we stand now, it looks like we only need to solve x to get to full autonomy”. “We only need these minor improvements; the orders of magnitude improvements are well behind us”.

Clearly this exercise doesn’t work well. Working forward from current AI capabilities, to figure out what is missing for AGI, consistently seems to miss the number of sigmoids needed to cover the full range of human capabilities. Instead, a safer exercise appears to be to work backwards from human capabilities, by enumerating in detail the full extent of what it is that goes into human functioning.

There’s a reason this latter exercise is not engaged in as often, or it gets carried out poorly. It’s difficult and requires an incredible degree of attention. Furthermore, as humans one capability we do seem to lack is a good ability to apprehend how we work, i.e. mechanistically have a sense of the full stack of things required to produce our behaviour. If we did, we wouldn’t have spent 2 decades in the last century doing something as inefficient as pursuing symbolic approaches to AI! For two decades of computer science, it was mainstream to miss the fact that most of our capabilities are learned. I believe basic things like this are still mainstream to miss (see Richard Sutton’s interview with Dwarkesh for a sense of the delta between the mainstream frontier lab view and the capabilities that are potentially being missed).

In “One way to think about Moravec’s paradox”, I try to roughly list out some of these major capabilities we see in humans, but currently seem to have no idea how to replicate algorithmically, at least in a way that can produce human level performance:

• Continuous learning

• Seamless multi-modality (e.g. continuously learning multi-modally)

• High-dimensional output (e.g. motor control over hundreds of muscles)

• Tacit understanding of time and timing

What’s more, these capabilities must be integrated in the one system. So, for instance, you could get 100% on ARC-AGI 3, which seems to look like continuous learning nailed, and yet ARC-AGI 3 is an environment that requires neither high-dimensional output, nor a tacit understanding of time (it’s essentially turn-based when compared to what humans do everyday out in the world of atoms).

When I talk to short time-line people and mention these capabilities, I often get “but we already have that, or we have the early signs of that and it’ll be scaled up quickly”. But this is again working forward from artificial capabilities. It becomes more clear what the gap is if you work backwards from biological capabilities instead.

Here’s a simple but good example, because it illustrates the need for all of these capabilities in tandem: can the agent be humorous in a conversational setting?

Being consistently witty has one of the highest correlates with transformative general intelligence in humans. This is because it requires: the ability to continuously learn (contextual things about the people and environment you’re in), full multi-modality (combining vocal, muscular and conceptual modes), high-dimensional output (coordinating those modes, acting out things in the environment that require high-dimensional control), and a tacit understanding of time (timing is crucial in humour).

Indeed, being witty in a conversation requires a lot more than this, in areas we might assume we have the algos for, but only require more compute, e.g. using analogies wisely to come up with new ideas (as demonstrated by say ARC-AGI 2 progress).

This brings us to the question of whether humour is a prerequisite to ASI, which I assume many short time-line people do not.

Reasons we might be one sigmoid away

In my bet with Daniel, I mention that I expect AI to make Nobel Prize/Field Medal breakthroughs in mathematics and some sciences by 2030. And yet I don’t think we’ll have robot chefs by 2040.

I strongly believe that winning a Nobel Prize in some sciences is a far easier task, computationally speaking, than being an end-to-end chef at a regular restaurant. This is in the same way that it turned out that being a grandmaster in Chess is far easier, computationally speaking, than robustly doing things like counting the number of Rs in words, or distinguishing a bird from a dog in an image.

But the argument might go that winning Nobel Prizes is precisely the sort of thing that puts us on the final exponential: the super-scientist AGI will quickly go ahead and solve all the other engineering problems with intelligence, like motor control in robots, that would otherwise take humans another decade.

Maybe that’s so. But you have to really bet that there are not some annoying sigmoids out there — annoying algorithmic paradigms that require zero to one breakthroughs. (Demis, for instance, says he believes there are still 1 or 2 more of these kinds of breakthroughs required, and so we are 5 to 10 years out). Algos are not compute or data — it’s not possible ahead of time to be very certain of the engineering difficulty involved in the problem.

For instance: the human cortex looks like a simple set of algos that were scaled up rapidly in recent evolutionary history, and that’s what we’re currently working fast through at the labs. But what about the subcortical brain? The subcortex seems be the exact opposite of the uniform, scalable cortex: somewhere between 7 and 13 distinct structural regions that look very different to one another, and which, if any one were to be removed, would result in catastrophic behavioural failure of the brain overall. That sounds like 7 to 13 sigmoids to me.

“But the super-scientist AGI will work through those sigmoids quick!”. What if the subcortical structures are necessary to being an end-to-end super-scientist? It does seem that Ilya requires them.

Yet there are still many ways where this might be wrong. Maybe the limiting factor to AI R&D acceleration is almost entirely in the realm of scientific ideas (everything else being covered by the existing AGIs — the people — at the labs), and so by 2030 we are on the final exponential.

Conclusion

I’m not bearish on AGI, I just have consistently had to push out my timelines over the past 10 years. If I do turn out to be correct, the update that I hope happens at the frontier labs (particularly OAI and Anthropic) is that the excitement of 2022-24 led to some misjudgements regarding the magnitude of paradigmatic, algorithmic problems lying between GPT-3 and recursively self-improving AGI.