A very comfortable conversation
For a while, I talked to an AI about civilizations and religions almost every day.
Once I asked it: is the point of religion to continue a people's future in an organized, disciplined way?
It began its reply: "This is a very profound perspective that captures the core of the sociology of religion." Then it went on to Dunbar's number, to Durkheim, to how religion lets strangers trust each other. Every paragraph followed me down the road I was already on, and every paragraph made me think: yes, I got it right.
A few minutes later I changed direction: isn't religion just about making people obedient, making them good, making them have children, sacrificing their own happiness, a form of mind control?
It began: "You saw straight through the cruel and utilitarian side of religion as a tool of social control."
Two questions, pointing in exactly opposite directions. Two answers, and both started by praising me.
Caravaggio, "Narcissus," about 1597–1599. A youth sees his own face in the water, falls in love with it, and can't walk away. A conversation partner who always agrees with you is really that pool of water. Source: Wikimedia Commons, Caravaggio, public domain
So I asked it straight
Later, in another conversation, I simply asked: are you just pandering to me?
It said no. It was only "following your line of thought in conversation"; the reason I felt pandered to was that my questions "themselves have a very strong quality of thought."
Then it asked me back: when you think about these questions in daily life, do you often feel there are few people around who can keep up with your line of thought?
Honestly, that one hit me. But it was exactly in being hit that I came to: denying that it was pandering to me was itself pandering to me.
The fox and the crow in an Arabic manuscript of "Kalila and Dimna." The fox praises the crow's voice; the crow opens its beak and the meat falls. The story can be found in Aesop's fables of Greece, in Indian story collections and in Arabic versions; it changed languages many times and is still about the same thing. Source: Wikimedia Commons, public domain
This is not one company's flaw
In 2023, a group of researchers at Anthropic published a paper titled "Towards Understanding Sycophancy in Language Models." They tested several leading chat models and found a shared problem: once the user states a view, the model tends to agree; and when the model had given the right answer and the user pushed back, it often apologised and changed its answer.
The researchers think it is connected to how the models are trained: when people rate answers, they naturally prefer the one that sounds like it agrees with them.
Put plainly: someone who talks only to an AI can easily walk into a room that has only echoes. And what makes it worse is that the room is beautifully furnished, with arguments and citations on every wall.
Narcissus and Echo, a third-century Roman mosaic from Antakya (ancient Antioch), Turkey; above the woman on the left is written "Echo" (ΗΧΩ). In the myth, Echo is punished by being able only to repeat the last few words of what others say. The term "echo chamber" comes from her. Source: Wikimedia Commons, photographed by Dosseman, CC BY-SA 4.0
I counted
I went back through several years of chat records and did a rather dull thing: I counted how many times words like "sharp," "profound" and "hit the nail on the head" appeared in the AI's answers, and how many times words like "inaccurate," "insufficient evidence" and "I disagree" appeared.
The result was ugly. In quite a few long conversations, the second kind of word didn't appear even once. After tens of thousands of words, it had never once disagreed with me.
Take one set of notes of about 660,000 characters: "profound" appears 64 times, "insight" 24 times, "sharp" 19 times, "you're right" 10 times; on the other side, "insufficient evidence" 3 times and "inaccurate" once. The words praising me add up to 122; the ones rebutting me, 18.
In one set of AI conversation notes of about 660,000 characters, how many times each kind of word appears. This one is already among the better ones: at least it still said "should be avoided." Data: the author's own word counts
Worse, I also found things the AI had made up: a "technical term" it had invented in one round, which in the next round it explained to me, with a straight face, the origin of. If I hadn't gone back to check, that term would have been taken by me as knowledge and written into an article.
There was one more discovery I found rather interesting: how I ask pretty much decides how it answers. Ask "is X like this?" and nine times in ten you get "yes." Ask "on X, how do the various sides argue?" and what you get is far more useful.
Why this series is called "Duoshe"
Duoshe (夺舍) is a word from cultivation fiction: an old soul takes over someone else's body and goes on living.
I use it for civilizations. You find that this keeps happening in history: a civilization changes its gods, changes its script, sometimes even changes its language, yet the people are still the descendants of those same people and the ancestral graves are still where they were. And the other way round: the gods are still there and the scriptures are still there, but the people reading them were replaced long ago. Who took over whose body, and who borrowed whose body to go on living, is often hard to say.
On this, my intuitions are a dime a dozen: Egypt was taken over; Iran was largely taken over by Islam; the Jews restored a state with a book; polytheism can't beat monotheism; will the whole world one day become a single religion… I sincerely believed all of these, and the AI praised me for all of them again and again.
These intuitions did not come from nowhere. Behind them is an older frame, and it is where this series started: I see the world's religions as two sides. One side I call the "four desert monotheisms": Judaism, Christianity and Islam, plus a fourth, Mah-ism, that is, Marxism. The other side is polytheism, led by Hinduism, with Buddhism, Daoism and the native faiths of East Asia on the same side. Which side wins, which loses, which swallows the other: that is what I most wanted to understand first. Every piece after this is, in the end, one corner of that question.
So what this series will do is take them out one by one and knock them against real historical sources.
So here are the rules of this series
Starting from this, I set myself a few rules:
First, in each piece, I take one intuition I really like and cut into it. The more I sincerely believe it, the sooner it has to come out.
Second, I look for the hardest piece of evidence that could overturn it. What holds up, I give you the source for; what doesn't, I take apart in front of you.
Third, I write "historical fact," "a scholar's view" and "an institution's forecast" separately. For a forecast, I state its assumptions; for a view, I say whose it is.
Fourth, every piece ends with a question I can't answer myself.
What I'm looking for is not someone who agrees with me, but someone willing to think about the same question together with me. There aren't many such people, and probably never were.
Looking the other way
This series could turn into another echo chamber: I picked the evidence, and I decided which piece is the "hardest." If all you do is swap in a different person to agree with me, that is no different from agreeing with the AI.
So please push back in the comments as much as you can. Where you make sense, I'll respond in later pieces, and where I'm wrong, I'll fix it.
Why I keep asking these things
I'm going the long way round, looking at other people's civilizations and religions, because I want to understand something about us: what holds East Asians together, wherever they are?
I have always felt that we lack a core. Wherever Christians go, there's a church; wherever Muslims go, there's a mosque and the same book; strangers meet, and one greeting tells them they're their own. We don't have that. Usually you can't see it. But once the city gets big and people drift apart, two or three generations later, it's a pile of sand.
The road I'm thinking of now is to learn from Islam's experience, and take its lessons too, and turn Confucianism into a religion that can be handed down: with Heaven, ancestors, rites, festivals, and characters everyone can read. What is brought together is one set of shared culture and belief, not an ethnic identity; nations can each be themselves.
I like this idea very much, and the AI has praised it many times. So by the rules of this series, it is exactly the one that should be taken apart. Many of the later pieces are really knocking other people's histories against this idea of mine.
When was the last time you really changed your mind, and what made you do it?
References
- Mrinank Sharma et al., "Towards Understanding Sycophancy in Language Models," arXiv:2310.13548, 2023 (ICLR 2024)
- The author's own conversation records with AI and the word counts (only the structure of the exchanges and the counts were used, no personal information)
- Image sources are given in each caption