Remy Grangien

One way to think about Moravec's paradox

Humans have a natural tendency to project intelligence, or ‘a mind’, on to a lot of non-intelligent things. For the longest time (and this tendency lingers) people saw minds in things like trees, volcanos, and statues.

Why is this? If you see the hint of intelligence in something — e.g. you hear the rustling of leaves in the forest, and hear it as a whisper — you tend to project onto it the only kind of mind you know of, i.e. the human mind. The forest becomes imbued with a spirit: it has its own memory, agency, and emotions.

We will get AGI and then ASI shortly, but before then, each novel expression of artificial intelligence will first appear, to the imagination, to have a complete mind of its own. People talk about ‘unhobbling’ LLMs: just remember that the researchers at the Dartmouth Summer Research Project thought a similar thing about the first computers. This is a human psychological tendency, stemming from our need to have a ready theory of mind for other people, while having close to no mechanistic model of the mind to check this reflex with. (In 2025, we have enough of a model to know that statues are not sentient, but not enough to know for sure whether LLMs are).

This tendency produces a seperate psychological phenomenon which pushes in the other direction. Because humans are prone to seeing sentience in everything, improvements in capabilities are normalised almost immediately. There is an initial wonderment — “AGI! Again!” — followed by a quick normalisation of the new phenomenon.

For humans, “Nothing is new under the sun”. You can trust that, if friendly extraterrestrial aliens were to arrive on Earth in 2026, by 2027 everyone would be back to their daily routine. This is partly because the concept of aliens and humans cohabitating was already present within our collective consciousness (indeed, for some, it would merely be a “I told you so” moment). But even if aliens weren’t an existing idea in our culture, the mind is quick to assimilate novelty anyway (think Cortez and the Aztecs, or any other colonial story).

Contrary to Daniel Kokotajlo and Leopold Aschenbrenner, I don’t think you have to be galaxy-brained or amenable to weird ideas to believe “superintelligence before 2030”. I think this will be the normie position soon enough.

2030 predictions

If I were to try and pick what is going to happen before 2030, it would be the following:

  • RL + transformers will automate a large amount of white-collar tasks that would otherwise take humans a working day, producing economic and productivity gains in excess of 1995-2005 internet.

  • Along with the roll-out of FSD, advanced personal assistants, and generative AR/VR, this will generate the feeling that we are in the grips of an impending singularity. This feeling won’t be confined to San Fransisco and the tech bros, as it is now. Instead, it’ll be a global, public market mania. (Compared to dot com or late 1920s, we are still nowhere close).

  • We will not actually get recursively self-improving AGI by 2030 — that’s not in the RL + transformer paradigm. Furthermore, we will not see >5% real GDP growth in the West.

  • Alongside massive geopolitical and sociological upheaval, the speculative AGI bubble will crash hard. (It will start looking like the beginning and end of the modern era, i.e. French/American and Communist/Fascist revolutions).

That said, it probably won’t be that much longer before we get AGI.

In Continual learning as a moat, I suggest that we do not have anything that looks like continual learning in AI (that pairs with SOTA), which seems to be one requirement for AGI. Today, I wanted to point out another architectural roadblock — the general rule that, without much higher dimensionality of input and output, we are unlikely to see human-level generality in AI.

Moravec’s paradox

Moravec’s Paradox is the observation that humans tend to be good at things computers are bad at, and vice versa.

One way to think about this is that humans are encoded genetically to be as general as possible: to go from foraging on the African Savannah, to building hundreds of thousands of tools that can then go on to build computers. Computers, on the other hand, are highly specialised to begin with, in the sense that they have way less compute than the brain, but can be readily optimised for specific algorithms that the brain sucks at, despite having heaps of compute, e.g. arithmetic, Monte Carlo tree search, etc.

How do you define generality? How do you define robustness? Well one way you can think about it is to say, “If I put this thing alone on a new version of Earth, would it be able to reach the singularity?”. Humans, as a species, clearly meet this benchmark (what the smallest number of N could be is unclear, but it’s perhaps close to a few thousand). LLMs do not meet this mark.

The criticism of this measure would be that AGI won’t need to be as general and robust as humanity to achieve recursive self-improvement, because the AGI will simply have humans do that difficult-for-AI stuff. I think this is a misunderstanding. That difficult-for-AI stuff is, I believe, one-and-the-same with what makes humans capable and powerful in the world.

The way I like to think about this — about why Moravec’s paradox is the case — is in terms of bandwidth of input and output. Let’s compare humans with AI for input:

Humans can take in around 10-100 bits per second of information while reading. The best LLM inference to date gets about 100x that, ~8000 bps.

That’s impressive, but what if we consider the central nervous system as a whole? The eyes, after all, generate something like 10,000,000 bps:

It’s not surprising that LLMs can do PHD-level mathematics, but still struggle with basic spatial awareness, and cannot do things multi-modally (like noticing the visual formatting error on the chart it produced).

That’s not to say that we cannot have AI today with gigantic visual input:

Why aren’t Teslas generally intelligent then? Well, they are still severely constrained when it comes to output:

The point being made here is that, for all three (input bandwidth, output bandwidth, and parameter count), humans are still ~2 OOMs above that of AI. If a model does have trillions of parameters and 1e25 FLOPs training runs, then it has incredible I/O bandwidth constraints, because they have been trained in the domain of language, and not high-throughout, real-time visual-motor I/O.

Thirdly, in the case of FSD, it’s true that output bandwidth is high, but it’s not equally high-dimensional. Ultimately, you can drive a car successfully with <10 buttons. Driving a car on the road is a highly abstract activity. Compare this with say the hand: what is the minimal number of end-motor outputs you would need to control the hand to successfully do a task like produce a coat from a pile of fabric, end-to-end? We are talking about 1,000s, if not 100,000s of “buttons”.

Think about the difficulty you’d have doing this if you only had one hand. Then think about the difficulty you’d have doing this if those two hands were not connected to multiple DoF arms, etc.

Once Optimus can perform long-horizon tasks, end-to-end, in a multi-agent setting, we will have AGI. (In all these robotic demos, notice that for tasks that take ~10 minutes for a human, the robots cannot deal with any serious environmental intervention, e.g. someone pinching the clothes from them).

As roon has pointed out, language gives us the densest compression of the world. But it is so information-rich precisely because it’s so compressed: to uncompress it, you would have to have on hand the same (or similar) compression algorithm. What is that compression algorithm? Well it likely has something to do with how those 1 quintillion synapses are trained end-to-end with the 11mb/s visual-motor system between the ages of 0 to 5.

A note on the METR eval

One reason agents don’t exist yet is because these models are doing their tasks ‘timelessly’. Any task that is turn-based, and requires no model of time whatsoever, is timeless. This can cause confusion, because you can arbitrarily improve a model of something that has no dimension of time, and then still fall on your face immediately when thrown into a real-time or multi-agent environment.

Another way to think about this: on the various math and science exam benchmarks, do we bother to ask how long a particular question took a human to figure out? We don’t (or we shouldn’t), because that’s not a measure of real-world task-time performance. We understand that the exam is asking instead for a timeless model of the world.

What is the difference between a math exam question and a software engineering task? It is not the case that one is one-shot, and the other requires something completely different called “multiple actions”. An action, like hitting an API endpoint, and getting a response back, is no different to breaking down a question into multiple steps internally, to get varying responses back from varying internal programs.

Think about the difference if the agent had to get some information from a live WebSocket, rather than a REST endpoint. (Hint: these are not agents).

How long it takes to build a factory, or a company, or a political career, doesn’t hinge on how fast you can do search on timeless problems. Of course, this can speed things up, if it’s the case that you are not failing on coordinating temporally between tasks and other agents, and you are acting coherently on long-time horizons.

Increasing the context window will improve the score on the METR eval. Increasing compute available will improve the score. Improving the RL, so the model has better reenforcement signals on how to use tokens better over the course of the task, will improve the score. But none of these things translate to “better able to coordinate resources in the real-world, in real-time, in a chaotic environment”. Is there a single agent model that could respond to a human-led DDoS attack, in real-time? What would that even look like?

But bro, Advanced Voice is real-time?

Not really. Ask it to leave 10 seconds before responding to you. Or to say one line of a poem, take a 10 second pause, and then say another line. These models have no tacit understanding of timing.

As a review of some of the roadblocks to robust AGI:

  • Program synthesis (CoT?)

  • Continuous/Online learning

  • Fully multimodal (none of this “But I told the helper tool to do the thing, let me reword that”, *somehow produces an even less accurate image*)

  • High-dimensionality, high-throughput I/O, especially during training

  • Tacit understanding of time (you can have continuous learning in an ARC-AGI 3 setting, while still being clunky and turn-based)