Ok so a short while back the Bay Area Rationalist community seems to have been revealed to be lousy with high-profile cults, most notably and convincingly, Leverage, which obviously makes me feel rather vindicated but also isn’t really what I want to talk about. Instead, what really struck me is a recurring fascination, whether openly discussed and a part of the dogma of the cult, as in Leverage, or discussed furtively without official sanction, as in MIRI, but in either case apparently quite common. The fascination is with what they call demons. Now, you might think that committed technologists in California getting sucked into weird occult cult shit is strange, but if you believe that I recommend reading up on Jack Parsons. Of course, Jack Parsons co-founded the genuine rocketry pioneer JPL, while the crowning achievement of Yudkowskian Rationalism still seems to be a Harry Potter fanfic and an extraordinarily large scam/cult incubator, but I digress.
By demons they don’t seem to mean quite the same thing as in the pop cultural or religious tradition, of a hostile supernatural (non-human) being manipulating or outright possessing people to do evil. Instead, what they appear to mean is a person, not even necessarily deliberately, getting other people to internalize their beliefs and thought patterns (these would then be referred to as demons). They call it demons though, because rationalists appear to be fundamentally opposed to being intelligible to anyone else, and also are massive nerds (for what it’s worth, one of their own favorite thinkers coined the term “meme” to describe something pretty close to this phenomenon already, though of course it’s not really usable these days I guess, and doesn’t allow you to develop paranoia about extremely ordinary social interactions). They also call it demons because they think this is an extremely bad thing, and not something that just tends to happen when you hang out with people for a while. I’m not here to make fun of people for being unfamiliar with social interaction, but the way many rationalists seem to have processed this apparently novel experience seems both unhelpful and in many cases actively dangerous.
I feel like there’s a lot to unpack with “demons”, really. I suspect I’ve observed the same phenomena as the Bay Area Rationalists did, and I’ve even been briefly distressed by them existentially, but I certainly did not become paranoid about being mindcontrolled by everyone I knew. Why is that? Well for starters I’m not in a cult. But there’s more to it than just that. There’s a particularly dangerous combination of intellectual rigor with self-evident, but wrong premises. See, if you’re both devoted to reason and to a particular set of premises, you will reason along with Sherlock Holmes, that "When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth." Of course, in reality arriving at an absurd conclusion is much more often an indication of some unexamined flaw in the reasoning, or the premises.
I’ve already explained that what these people seem to be experiencing as a novelty is something which most people are used to as a basic feature of social interaction. However, this does not itself suffice to contradict their claims. In fact, a lot of people who are entirely used to adopting beliefs, mannerisms, or entire ways of thinking from their environment could stand to do at least a little more thinking about the process. So if we ignore, to the best of our ability, all the cult stuff, and as much of the untreated (or unsuccessfully treated) mental illness as we can, what can we try to reconstruct of the chain of reasoning here, and actually respond to the argument. I will be venturing into quite a lot of conjecture here, but I think I have a pretty good idea.
We start with the premise (though because it is such a fundamental premise, they likely don’t usually think of it as such) that there are such things as individuals, and that we are those individuals. An individual is a fundamentally unique entity (perhaps we could call it a unique string or pattern of information), separate from other individuals and the world, that interacts with the world as a coherent whole. Next, we engage in some self-reflection, and we compare our image of ourselves with our experiences of ourselves. We discover that they differ in important respects, and that often, in our experiences of ourselves, we find things that seem more like our images of others than of ourselves. Perhaps they are simply verbal tics, or turns of phrase, perhaps they are something more significant. This is, in any case, a violation of our individuality, manipulation, a hostile act, an infection. A demon. That’s roughly my interpretation of the thought process, simplified a bunch.
If I’m right, and this sense of disrupted individuality is at the core of this demon paranoia, then I think the people suffering from it are the unfortunate victims of unexploded ideological ordnance from the Cold War, either capitalist Cold War propaganda or 90s triumphalism taken to unfortunate extremes. See, the central narrative of the soi-disant First World was that, with their democratic governance and free market economies and freedom of speech, they understood the importance and the rights of the individual, whereas the communist states were a homogenous and terrifying mass of insects, or endless ranks of individuals, suppressed but yearning for freedom, or both, depending on the situation. The US likely has the most of this propaganda floating around in the environment, polluting political discussion with libertarianism and other preposterous nonsense. Rationalist communities in particular seem to have a lot of members who are still in their libertarian phases or are carrying serious baggage from them, specifically including the exclusive focus on the individual.
So what’s so wrong with strict individualism as a premise? Well, it’s just not true. Yes, everybody is unique and everybody deserves respect and such. But that is not incompatible with recognizing that we are all but mosaics of mosaics, that we are formed by our experiences and relationships. And not only in good ways. No man is an island, entire of himself, for both good and ill. We construct our selves, sometimes in opposition to, sometimes along the pattern of our social environment, but always in the context of it. We live in a society.
I can, from experience, anticipate a likely objection to the low importance I assign to individuality. Surely I respect people’s rights, and wish to protect my own? And how can I defend the rights of individuals if I deny their existence? In fact, the so-called “individual” rights are misnamed, and the concept of individual, even though I do not mean to abandon it, is by no means necessary for them to be applied, at least if by those rights we mean, and this is generally what is implied, freedom of speech, thought and assembly, of political participation and various others besides. Because if any of those grand rights are to be rights in the first place, they must apply to the society at large. If they only applied to a particular subset, to some individuals, then they would be nothing but the privileges of a particular class. So they are not "individual” rights, but the rights of a society, and also its duties to itself.