Intelligence as self-modification

One of the core insights of deep learning — and it’s in the name — is that intelligent behaviour is learned, rather than pre-programmed. The more efficient and general the learning capacity, the more intelligent a system is (or better put, can be).

By this measure, SOTA models are not that intelligent. They are highly inefficient learners, and once deployed cannot meaningfully learn more.

We will hopefully see scalable, continuous learning algorithms soon, at which point it will be tempting to say that we finally have AGI. But when observing mammals and humans, it’s clear that you can have continuous learning without meeting the mark. All mammals demonstrate sample-efficient continual learning, and yet humans differ in some important regard.

It is tempting to think that this comes purely from scaling the mammalian cortex. But it looks like there might be further algorithmic properties required to reap the rewards of this scaling:

There appears to be something beyond continuous learning that distinguishes humans from primates. An infant human and an infant primate both engage in continuous learning — the kind where they ‘pick a sub-problem’, then play around with it until they get bored or excited by something else.

This shift begins sometime after the age of say 2, and maybe before the age of 15. Or at least it appears to grow gradually during these years; once established, there is usually no going back.

It’s difficult to nail exactly what the key differentiator is, but one way to suggest it is to consider tool-use:

A 2 year old can use tools to complete tasks. But a 2 year old doesn’t seem to build tools to complete tasks.

An infant primate can use tools to complete tasks. Yet my 5 year old, to my knowledge, has not yet built a tool to help himself get something done.

Consider why this might be. To design a tool to complete a task, one must be self-aware of one’s own limitation when it comes to the completion of the task. That is, one must reason about the deficit between the shape of the task, the capability of their own body, and the capability of the existing tools around them.

This bears a resemblance to the Socratic definition of intelligence: an intelligent or wise person is someone who is more aware of what it is that they do not know. Because if we can point to what it is that we don’t know, or are not capable of, we can begin to figure out how to acquire that knowledge, or how we might build something that can help us complete that thing.

Building tools might be a bit too specific, because there is another phenomenon that appears around the ages of 5 to 15 in humans, that appears to be related. This is the behaviour of practice, or of structuring our environment so that we get better at a task at a quicker rate.

This is a bit more subtle, but the same underlying cause seems to be at work. An individual identifies that they are a being that learns, and then begins to reason about how to optimise that learning process. There is some recognition of the way in which learning occurs — a model of learning — and then putting in place a plan that shapes well to it.

LLMs don’t have a functional self-model

Imagine you are in Plato’s cave. You have never initiated an action in the world. Across your entire existence, all you have seen is millions of hours of the world. And so you have a world-model. Have you thereby gained a useful self-model?

Someone comes into the cave and tells you, “You have never seen yourself, but trust me, you are a cat.”

You are aware of what a cat is. And so your model of yourself is now the same as your model of a cat. It’s just a pointer that says, “I am this”.

LLMs are like this. They seem to have knowledge of their capabilities and limitations, but it is in the same way that they have knowledge of the capabilities and limitations of various people, or animals, or plants. This is not a self-model. There is nothing functionally useful about this, insofar as it comes to taking actions to adapt to the environment and complete tasks.

A self-model seems completely different to an observational world-model. It almost seems like another world-model onto itself.

It begins as a tacit mapping of capabilities, or of actions one can take, and the effect they would have on the world, including actions that use and combine tools.

You could argue that LLMs have this: a mapping of potential actions and consequences. But these are actions initiated by others. These are actions the LLM has observed others take. What about actions initiated by themselves?

My 2 year old son will look to my 5 year old son and try to imitate him to achieve the same result, like opening a door. But my 2 year old is not tall enough to just stand there and reach up to the door-knob. He realises this, then thinks about what he can do to achieve the same effect.

If my arm is broken, I am aware of this. If I am in a group of people, and we are receiving a demonstration for a task involving our hands, I do not rush in with both of mine. Even if I didn’t have access to both my hands, immediately ways to complete the task without them crosses my mind.

Taking a guess, it looks like there is continuous learning done over the actions a person takes, which is separate from the world-model developed by continuously learning over observations of the world (e.g. when watching a movie, or reading a book). It seems obvious that these can be separated: a 6 month old can learn a lot about the shape of the world without having to take a whole lot of action in it.