Censor-y Deprivation

Some thoughts about LLMs

A bit of news to start us off: This is How You Lose the Time Wars is a Kindle deal of the month in the United Kingdom—available for 99p! I believe it should be available at other electronic retailers for the same price, due to the wonders of price matching.

---

In Roland Allen’s The Notebook: A History of Thinking on Paper I encountered this delightful aside, which really should be the foundation of someone’s Scholastic Book Fair bestseller: the ship’s log of the 19th century whaling ship Fleetwing was kept by the captain’s 14-year-old daughter, “who covered the weather, the ice pack, the ship’s catch and her needlework in equal detail.” (pg 323) Ugh! Dad just nailed a gold coin to the mast and promised it to the first person who sited his vengeance-whale, why doth he always have to be so extra??

---

I’ve been sitting on the following essay for a while now; it’s one of those ideas that struck me in the middle of a run and sometimes I feel the central idea is so obvious that everyone who’s paying attention must have thought of it six or seven times already. But Vox’s Future Perfect newsletter ran an article this morning about how good “AI” systems have become at geo-guessing (identifying where a photo was taken even if it doesn’t have any obviously identifying information), and the implications this fact has for digital privacy—which made me think, well, perhaps this is worth saying. Henry Farrell often argues that LLMs are best considered not as ‘artificial intelligences’ in a science fictional mold, but as social technologies for organizing large amounts of information—it’s the dark potential that concerns me here.

---

In 2008, there was an uprising in Tibet. Religious protesters were arrested, tear gas was fired into crowds. Protest and counter-protest built into riot. The PLA marched into the city. I was living in southern Anhui province at the time, teaching English at a rural school. Chinese Central Television carried images of Lhasa burning.

One day I came home from lunch to find the wall of our campus apartment covered in leaflets, “Tibet is an indivisible part of China” scrawled in red paint across our front door. The school administrators were (sincerely, I thought) contrite; they claimed it was the fault of some overeager kids, who would clean up and apologize. The paint was scrubbed away, the leaflets taken down. I remember a stammered apology from some kids I didn’t know, who may have been the perpetrators—or not. Hard to say.

At the time, I was sending regular updates to friends and family about my life in the PRC. I wrote about what had happened. I sent the email. I caught a bus to meet my partner in Shanghai. I tried to check my email in the youth hostel—and found that I couldn’t. My inbox wouldn’t load.

I knew about the great firewall, of course, but I’d never run into it so directly before. It wasn’t personal: someone in the Public Security Bureau (or wherever) had set up a filter to block all sites that said “Tibet.” To be able to access my email again, I had to VPN out of the country, log in, and delete the message I’d sent. Once I did that, I could access mail normally.

I haven't messed around with LLMs much. They don't appeal to me and at the moment I don't need to do anything with which I’d trust them to help me. I also have a chip on my shoulder about the entitlement of the people who make and boost and sell them: give me all human work, says the Man in San Fran, so I can make God and profit from it! (The promises of the Man in San Fran with regard to making God should be treated with the same skepticism as any of his promises. If you want to convince people to give you infinite money, you need to promise them at least a ten percent return.) It’s not the tech’s “fault” that I don’t like its boosters, but they do leave an aftertaste.

I do, though, think it’s a mistake to dismiss the technology. Teachers are heading back to longhand in-class essays. Any image search turns up page after page of Midjourney slop. I run into folks—book readers!—who say they’re learning about subjects by asking ChatGPT about them—which, I don’t know, I wouldn’t. But it’s illustrative. People claim they’re using LLMs to write office emails, which seems innocent enough, though I suspect outsourcing basic social problem solving like, "how can I politely rephrase Frank's request to Marketing so he gets what he wants without starting a huge fight," will reinforce the sarcopenic tendencies of modern knowledge work, aiming straight for the Wall-E hoverchair end state.

But none of the chatbot / auto-dungeon master / digi-secretary mishegoss touches on what friends who do understand the technology have told me LLMs are really good at & useful for: sifting through huge amounts of unstructured data in situations where a fallible best guess is better than the alternative, and there are few consequences for false positives and negatives. Looking for patterns in massive piles of digitized medical records or journal articles, for example. Finding connections, surfacing truths we have technically recorded, but which were invisible.

That sounds well and good and promising in the long run: maybe some good will come of all this faffing about. But then I think back to my experience with gmail in 2008.

One of the great barriers to broad-spectrum internet censorship, is the sheer perversity of the human mind. It's easy to blacklist particular words—but try that on the internet and see how far it gets you. On the Chinese web, many years ago, the word for "harmonize" was used as a euphemism for "to censor." So the censors started censoring posts that talked about "harmonizing." So, for a while, users would, instead of saying "harmonize" or "censor," write "river crab," which in Chinese is a rough homophone of "harmonize." I imagine "river crab" was promptly censored not long after, tough luck to the river crab hobbyist community, but you get the point: an arms race between the blacklist and human weirdness, deforming (or transforming) language at hyperspeed. You've seen the same force at work if you've ever wondered why someone on TikTok talks about "unaliving" an enemy in a video game. Can't say "kill" on TikTok.

But an LLM can censor differently, by scanning incoming text, determining to what extent it is doubleplus-ungreatagain or whatever, and blocking it (or shadow-blocking it in a social media context, turning down the post’s “social reach” until it might as well have never been made). Censorship is a domain where false positives don't matter, where it's okay and even helpful (from the perspective of the system-owners) if the system gets it wrong. Unpredictability is an advantage for a censorship regime—by variable enforcement you encourage people to run a version of the censor in their own head, more restrictive and far-reaching than any actual censor. Remember the parable of the rats and the lever, in which giving the rats a food pellet every time they press the right lever leads to fewer presses of the lever overall than giving them a food pellet, say, 25% of the time. (Parable might be the right word here, I went hunting for an academic source for this & found a lot of bloggers and designers and such talking about this effect, but the actual science seems murkier.)

In the D.O.G.E. and NSF grant crises, we’ve already seen censorship of broadly this sort—culls of grant applications flagged as undesirable to the administration because they investigated “diverse gene expressions among zebrafish” or “equitable distribution of resources among ant populations” or whatever. The censor doesn’t mind accidentally censoring things that “should” be okay—the uncertainty is the point.

All of which casts an even more sinister light on reports that some large, well-resourced actors are running around creating large websites full of apparently human-generated text specifically not designed for human consumption, but bot-readable in the extreme—attempts to to influence LLM weights toward the creators’ ends.

We’ve seen how LLMs can enable or support the large-scale astroturfing already so prominent on the modern social web—consider the book ban “campaigns” that trace back to one disgruntled individual with a computer and too much time on their hands. About ten years ago in the West, you could rely on a skim of social media to get a sense of the overall “vibe” within a particular language group and a particular time zone (at least among people who use social media)—I don’t think that’s the case at all any more. It’s easy for small highly motivated groups (activists, fash, marketers, spooks) to produce something that looks like a groundswell of sentiment, drowning out “organic” noise. Groups that have come to depend on social media as a windsock of public sentiment (artists, journalists, academics, marketers, spooks) may find themselves, broadly speaking, kinda fucked.

Add to this an LLM-based censorship regime—either built by out-and-out authoritarians or out of an understandable desire to do moderation at scale on the cheap, without subjecting actual humans to the burnout and depression that comes from a job wading through the collective muck of human evil—and you’re headed for an operationally useless public internet, a “dark forest” where no one speaks for fear of being heard—or, since silence is itself a message, where everyone speaks but they speak only… what? Not things ‘everyone will agree with,’ but a sort of rootless emotional chatter. Umbrage and despair, not anger or action. It’s a dystopian vision to put it mildly. But if you want a glimmer of silver lining to all this, remember that while suppressed speech is comfortable for authoritarians, it’s not actually that good for them. Regimes need an accurate sense of what their people want—so they can seem to provide it, so they can direct popular anger against regime targets before it shifts against the power structure. I’m told the PRC used to have extraordinarily comprehensive and clear-eyed internal polling. I’d bet they still do, I’m just not in touch with the on-the-ground over there these days.

We may be living in the aftermath of this shift already. Unlike predictions of hyperintelligent GenAI paperclip maximizing time traveling demons, this sort of thing is possible with presently-available tools (if you had enough compute to throw at the problem, which is probably the limiting factor). It's an unsettling thought, though, and if you understand the tech more than I do (& I know some of you do!) I would be interested in hearing whether you think I'm off base.