I sat down with a group of students and recent graduates from Finland a few months back, and one of them asked me: “I am very concerned about AI alignment. It feels like a superintelligent superpower ruling over humanity is not being taken seriously enough. Why isn’t society taking it seriously, and how do you think we should approach it?”

It took me a second to think about the first part of that question, which I hadn’t deeply considered before. My answer then, and my hypothesis now, is that there are a few modes of people who are not concerned about alignment: the first mode of people who know a lot, but don’t buy it: in those cases, they realize how hard it is to change culture through technology, how hard it is to get software adopted at scale, how expensive compute is, how inefficient capital markets are, how much drama the big research labs have, et cetera, et cetera. Many of them are techno-optimists in their self-image. They believe technology is the tools we use to solve problems at scale, and that we are going to be too busy solving human problems to get gummed up in any of these big questions. Embedded in this techno-optimist view is actually a deep skeptic: it seems like they don’t functionally believe we will be able to build a massively paradigm-shifting species-transforming AGI or ASI, so what’s the point in worrying about it.

The second mode of people is those who know a lot, and are excited for it: in these cases, they welcome cyborgs, consciousness uploading, chasing immortality through AI-powered biotechnology, and so forth. They seem to believe that technology is inherently good, over the long run. But technology is neutral! One can’t think of a hammer as good or bad — or the wheel as good or bad. They are useful, but plenty of useful things are useful to harmful ends. Those who welcome the future prima facie are, I reckon, blocking out part of the implication of technology.

And then, of course, there are those who don’t know a lot and don’t buy it.

Of these three groups, the middle group seems the smallest (by far). In the tech community there is a lot of chatter about how cool the future will be, but in playing the implications of our current trajectory out 20, 25 years with any rigor, I suspect many arrive at the techno-skeptic (first group). It’s easier to just think about next quarter, or next fundraiser, or next model release cycle, than to deeply think about what the future looks like.

On the second question, I felt more prepared for my answer. Imagine describing AI to an ancient human –– “a superintelligent invisible being designed by the body of all of humanity’s recorded expression that helps us be productive, less lonely, and guides us through work and personal life.” –– almost any person in almost any civilization in the world would shrug and say “spirits, angels, devas, dybbuks, gods. Sure, no big deal.” We have, in fact, 100,000 years of robust and time-tested systems of organizing our society based on the belief that there are higher powers than ours. These higher powers move among us, determine how we all should act, and with whom we should be in communication — this is older than just about any principle we have together. Humans as the highest power in the universe is a relatively new cultural belief (and frankly, it is rare — 85% of the population¹ practices a faith that assumes a belief in God), and religion has withstood wars, technology, epistemological and scientific revolutions.

We will ascribe worship, prayer, and other religious practices to AI-systems the same way we have to Earth’s other natural systems over time, and some faith practices will call AGI/ASI the very spirits and gods they’ve been telling you about² — and in some cases, waiting for. And as for the alignment question: the Christians call it “fear of God” for a reason.³. If you want to understand what our society will likely look like in a world with AGI/ASI, look at the sociology of religious societies. Read the history of pre-scientific civilization. On the one hand, you may be surprised that it will be less different than you think. The Enlightenment hardly dented religion worldwide. It just reallocated (some of) it. On the other hand, if you aren’t already deeply immersed in faith practices, already traversing in communities who speak of God and the spirits, it may come as a shock to you. Get ready for a return of prophets, messiahs, medium-channelers and voodoo masters. Buckle up for another major a reallocation of the major religions, at the scale of the Protestant Reformation, if not Emperor Constantine legalizing Christianity in the Roman Empire. The mystics will have their day in the sun again. Fortunately, we’ve seen this movie before.

1. https://www.pewresearch.org/religion/2022/12/21/key-findings-from-the-global-religious-futures-project/ ↩︎
2. What this doesn’t contemplate is whether the ancient religions knew it was coming. Deeper, more mysterious question. ↩︎
3. NB “fear” of God in Christianity is more like the Greek word phobos which is more awe, than terror. Think about your first Waymo ride, or your first very-human voice-AI demo. ↩︎