Palantir as the next frontier lab
If experience is the limiting factor to the cultivation of useful intelligence in the world, then Palantir appears to be the organisation best placed for the cultivation of artificial intelligence. Anthropic — in a world where nobody trusts them or lets them have access to interesting parts of the world — seems to be out of the game.
Would you rather have another standard deviation in IQ, or twice as many of yourself?
In the rationalist worldview, the answer to this question tends to be more IQ. Intelligence is power, and intelligence is something that correlates strongly with IQ. Elon Musk can do what he does because he is intelligent. The average person cannot do what Elon does, because they are not as intelligent. Clearly, the difference between Elon and the average person stems from their genetics, IQ is hereditary, etc.
But here’s a different framing. Would you prefer to have an IQ of 140, 10,000 years ago, or an IQ of 100 today? Which individual is to be considered more intelligent?
“They train Agent-2 almost continuously using reinforcement learning on an ever-expanding suite of diverse difficult tasks: lots of video games, lots of coding challenges, lots of research tasks. Agent-2, more so than previous models, is effectively “online learning,” in that it’s built to never really finish training. Every day, the weights get updated to the latest version, trained on more data generated by the previous version the previous day.” — January 2027, from AI 2027
If it’s not already obvious, this where we are at now. Is it all over? Anthropic and OpenAI have an internal model that is online learning from millions of user sessions, concurrently. Is this the superhuman RSI?
I’ll try to steel-man this position. “The world is this machine, the mechanisms of which the intelligent mind can discover, and use to its own power and advantage. Einstein and JVN were very intelligent people, and so could discover facts about the universe that would prove to be powerful. The internal Anthropic ASI agent will know more in mathematics, physics, psychology, etc. than any particular human, and use this to one-shot the economy, politics, etc.”.
There are many different critiques that come to mind, but the one I would like to focus on is: why is there just one agent in this story? How is it possible that a single agent one-shots everything?
“Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis — perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration — by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.” — Yudkowsky, “That Alien Message”
This is the worldview AI 2027 and Anthropic come from. The reason it can be just one agent is because the universe is transparent in such a way that a single superintelligent agent can figure out or saturate the list of facts about the world very quickly. Individual humans bob around just beneath the limit in some places — say Einstein in physics — but superintelligence quite quickly gets up his level, then blitzes past and snaps together the last pieces of the puzzle in all sciences.
Coming from the other direction, suppose most facts about the world are not learned through superhuman sample-efficiency on a very small amount of data that’s already out there. Suppose instead that there is some limit to sample-efficiency that the human mind sometimes, but rarely approximates, beyond which the sample quality — the data — becomes the limiting factor.
I don’t want to make a grand claim as to what the actual structure of the world we live in is. The point here is to say, if it turns out that the rationalist clique running Anthropic turn out to be wrong about near-term RSI, it will potentially be because of their own sense of what the actual structure of the world is.
And if Anthropic is wrong, the anti-thesis to that worldview happens to be the worldview found among the people in and around Peter Thiel and Alex Karp. Hegelian, sceptical, empirical, politics-over-science, etc. The world is an idealist mystery, not a rational object. The anti-thesis of Bay Area rationalism.
I assume it wasn’t the plan, but this also points to why Palantir might outdo Anthropic in the development and application of AGI.
It is difficult to identify what the exact crux is between these two worldviews — or at least a way to technically explain the crux, i.e. a technical explanation for why future AI will look the way it does.
Firstly, to pin down the hand-wavy crux: is the world fundamentally mysterious, or transparent?
I think in their own way, the rationalists have encountered the mysterious world: no matter how much brain-power has been directed at the alignment problem, there is no way a priori to solve it. Indeed in so many ways, we are less sure than ever about a solution to the problem, despite knowing so much more about it. How can that be? How can more time spent on the problem lead to more confusion?
To solve the alignment problem, you have to learn some things. Okay, so what do you learn first? Neuroscience, psychology, decision theory, machine learning, sociology? Okay, so you think neuroscience is a good place to start? Counterfactually, suppose that it isn’t: at what point do realise this and move your research agenda over? When do you make that decision, and why? What does success look like? Wait, you’ve done some research and now you realise that what you thought was the alignment problem turned out to not really be the actual alignment problem? Okay, start again, etc.
The technical explanation for a mysterious world would be that intelligence is developed through data, you generate data by interacting with the world, however you have no way to know with certainty what data you should generate. (This also goes for producing outcomes in the world: you come up with some plan, you have limited intellectual resources to try make it work, but you make that decision to commit to it, and then you see if it works out after the fact).
At a high-level, the agent inside Anthropic right now is a product of what data the guys at Anthropic thought it was most useful to train on. Were they right? Were they wrong? We’ll see. Will the internal agent go on to collect the right data — the right experience — to continue on the victory path? We’ll see.
Anyone that says it is obvious what to do — these are your rationalists. Anyone that talks with a great deal of uncertainty — these are your empiricists.
I find it is more difficult to explain the empiricist worldview — particularly those in and around Peter Thiel. If there is a Bay Area epistemology, there is a Peter Thiel epistemology that is world’s apart from it.
If there is one deadly sin in the empiricist epistemology, it is to not step into the mind of God. Of course, this is the defining feature of rationalist thinkers throughout history.
“I spend a lot of time looking out through the eyes of superintelligence.” — Yudkowsky, “Eliezer”
What does the philosopher do then, if not step into the mind of God? Remembering that both Yudkowsky and Thiel have presented themselves under that title before — Philosopher.
The empiricist sets out into the world, gets their hands dirty, and bit by bit tries to unravel some small corner of the universe, not through reason, at least not primarily, but through experience.
(There is of course something here about the difference between realism and idealism. There is another story about how the facts about the world are not to be discovered, but are generated by intelligence. This is obvious enough to some, yet seemingly a mystery to Bay Area rationalists).
The empiricist is concerned with learning about the world in front of them. The rationalist is concerned with learning about some reality that may or not exist beyond the world in front of them. Yudkowsky — but also Russell or Leibniz — would claim they are learning something important about the world through reasoning about it; the empiricist would say they are learning precisely nothing. Indeed, they are convincing themselves of things that are precisely refuted by reality. Don’t step into the mind of God, because you are not God. God is a logician? Now God is a mathematician? Now God is an artist? Isn’t it peculiar that the philosopher always finds that God shares their thoughts and habits?
It’s not that rationalism is considered ineffective under the empiricist epistemology, but that it is considered actively harmful. Rationalism often seems to lead people far away from learning about the world. Instead of gathering experience about the world as a mortal human, you push experience aside and reason with the immortal (whatever maximising for reason is supposed to mean — it has changed dramatically over the years, but typically looks like using mathematics or logic to improve or correct a biased human cognitive structure).
This is where it gets interesting with Anthropic and Palantir. Anthropic, founded by Bay Area rationalists, so far has been chasing reason, while Palantir, founded by Peter Thiel style empiricists, are maximising for experience (or at least it turns out they’ve put themselves in a good position for it).
The variable part of each scaling paradigm is in the data. That is the core insight of deep learning. Intelligence is a model of the world, and the capabilities of the model are a reflection of the shape of the data it receives. My call is that the data Anthropic should want is currently accessible only by Palantir. And because compute and ML talent is fungible, and increasingly accessible via decreasing costs to technical intelligence, Palantir could be ready to take Anthropic’s place as the organisation with the most intelligent agents.