Continual learning as a moat

Dwarkesh Patel recently argued that ‘continual learning’ was at least one thing that seems required for AI to reach the standard of ‘AGI as useful as a single friend or employee’. Dario Amodei, when asked about this, vaguely pointed to the bitter lesson for why this doesn’t seem like a major concern or setback.

(There was a small splash a month back when it looked like a lab had made a breakthrough in this is space. Elon Musk got Grok to dunk on it — it doesn’t look legit).

I would like to make two points about this. The first is to suggest that ‘continual learning’ will likely become a focus of the frontier labs in the coming years. The second is to suggest that this will not be some emergent property that pops out of scaling, but that it will likely require a paradigmatic architectural shift, above that of CoT.

1. Labs have a strong incentive to make continual learning work, and probably soon.

The reason for this is two-fold. The first goes to Dwarkesh’s personal observation, which is that lack of continual learning is at least one reason he cannot replace salaried employees with AI. If continual learning becomes a thing, suddenly we see full-time jobs, like video or transcript editor, replaced by yearly AI subscriptions. It would be the thing that takes personal AI expenditure from ~$30/month, to $300/month.

The second reason is that continual learning would make any particular AI extremely sticky. Presently, there is close to zero cost to switch between ChatGPT, Claude, Grok, or Gemini. Building an AI into a device creates a small moat, but enough: just download another app, or buy a competing smartphone next time. The same goes for companions, where the only stickiness is the attraction a user learns towards the fixed character. But with continual learning, the other thing happens: the AI learns towards the individual; it moulds itself to them. The better the continual learning, the better the moat.

Presently there is a race to make base-models as capable as possible. If we hit diminishing returns in what that can accomplish in terms of utility to the user, then continual learning will become the new paradigm to scale along. (In Dwarkesh’s case, because it increases the competence of the AI on practically important jobs he employees for; for the average consumers, it increases the emotional and day-to-day utility of the product).

2. Continual learning does appear to require a larger architectural shift than CoT, and then a whole lot of engineering work beyond that.

There is a sense in which LLMs do continually learn today, which Dario points to. OpenAI and X collect millions of reinforcement signals every week, and use these to continually update the weights of their core model. But this is not what is meant by ‘continual learning’ — it’s not enough to scale and speed this process up. After all, using user input for RL (both collectively, and then fined tuned individually), has been a thing on Youtube and Facebook since the mid 10s.

The claim here is that for an AI to be massively impactful in my life, say as my 24/7 personal assistant, it would have to be fine-tuned on certain details about my life that are not accessible at a distance, e.g. by learning over the distribution of everybody else’s lives.

A model that cannot learn continuously, for instance, does not have a spatial model of my house. It doesn’t have a model of my personality. It doesn’t have a model of my friendships. It doesn’t have a model of my family. It doesn’t have a model of my daily commute, and how that relates to where my career is at. It doesn’t have a model of where my career is heading. It doesn’t have a model of practically anything that is of particular relevance to me as an individual.

There are of course plenty of services I pay for where I am accessing a general model of something, i.e. cardiology, with some low degree of personalisation, e.g. a specialist health appointment for some specific cardiological issue, given a echocardiogram. But the high-bar for an AGI assistant is whether it can do for me what a full time salaried personal assistant could do, which is make my whole day dramatically easier. This sounds like a high-bar, but it’s not when you consider what qualifies as a transformative technology in recent history: electricity, cars, smartphones, self-driving cars, etc. It has to be something that makes life overall easier, and this is indeed the target the frontier labs have set.

CoT and ARC-AGI 3

CoT is in the direction of an architectural answer to Francois Chollet’s criticism of 2023-era LLMs. I say ‘in the direction’, because ARC-AGI 2 is an incredibly basic (i.e. low-dimensional) test of whether an AI can do something like program search (generate insight given a novel situation). But it was clear with o1 and o3 that some kind of program search was now beginning to occur, and a whole lot of engineering work would be required to scale this up to program search in more complex domains (first mathematics, then perhaps biology, psychology, etc.).

ARC-AGI 3 is perhaps a test for continual learning. When playing a game you have never seen before, you need to in real-time learn a model of that game. You need to get better at the game as you play it. When Claude plays Pokemon, it’s not doing this. It is repeatedly referring to a fixed model it has of the world, and the game of Pokemon within it, and it can only adapt that model to the specific game instance within its context window. The underlying model of the game — the weights associated with it — do not change.

At a thousand foot distance, this seems like a much larger architectural jump than CoT. CoT is doing something on top of the weights that are already there. Continual learning likely means applying permanent updates to underlying weights in real-time, in response, ideally, to continuous data. Furthermore, what data is received, i.e. what actions the agent takes in anticipation of what it will observe, has to be related to some idea of what data would be best for training! That seems challenging, especially in contexts where you have to learn fast (i.e. you cannot spin up a million instances of yourself, and a million instances of the particular situation you are in).

Per More on agency, if you pay attention to how well humans function on long-time horizons, in contexts where there are multiple intersecting tasks, on such little data, and with such a high degree of input and output dimensionality — suddenly current transformers appear incredibly basic, architecturally! This might be one reason why labs will maintain and verbal interest in maxing out ‘scientific insight’ via program search, before looking to continual learning, even if it would propel consumer AI models into a $1T product-class.