A simple hypothesis for why ASI cannot run the world in 2030

The classical virtues — leadership, courage, wisdom, patience, forgiveness, open-mindedness, focus, etc. — are not contained in the training data. There are descriptions of them, but they are trained through acting and reflecting in the world over a lifetime.

They are also not unlocked through brute force. In what is one of the most common lessons in life, deprivation often aids rather than hinders the development of the virtues. A simple way to deprive a person of the virtues is to deprive them of a challenging life. (Though this is not to say that AIs will necessarily have easy lives).

You can imagine a world in 2030 where we have a single agent more intelligent than any particular human at any intellectual task. But I cannot imagine by then an agent more virtuous than any particularly virtuous human.

When I talk to RSI-pilled researchers, this appears to be a crux. They can imagine a more virtuous agent, perhaps because they do not fully recognise human virtue, its use in the world, or where it comes from.

---

What are the lessons to be drawn from The Brothers Karamazov? What are the lessons to be drawn from a random Mr. Beast video? If you can tell me, then maybe ASI will run the world in 2030.