The Harm Of 10 Years Of AGI Doomsday

Started: 2023-04-02 Last Modification: 2023-04-02 , 4717 words, est. reading time: 23 minutes

FLI published an open letter, led by like Elon Musk (of “apartheid emerald mine bootstrapper” and “burning 54 GigaDollars to own the libs” fame) and also some people I respect like Yoshua Bengio, calling on labs to pause research on large language models, and specifically to pause increasing the scale of them. Having followed the discussion in then field of “AI safety” since about 2013, I conjecture that there is a lot of heterogeneity in the motivations of the signees - some are gonna be worried about how we use these models, some will worry about the AI going FOOM and turn us all into paper clips and Elon Musk I’m gonna assume is just salty because he’s not gonna get a cut of the OpenAI 100x “limited profit”cap¹

A day later, LAION launched a petition calling for the establishment of the “CERN of AI”, a public institution to research open source ML models, AI safety methods and, crucially

100’000 state of the art AI accelerators

with a public access model in order to allow people to actually train models toe to toe.

The response, as of 2023-04-02 09:06 is as follows:

• FLI letter: 2825 signatures
• LAION letter: 1090 signatures, including mine

The reason for this post, however, is that both in the discussions I’ve had so far and on the petition website itself, people are arguing against the establishment of the center on an “existential risk” basis: AGI has an XX% risk of destroying humanity (between 10 and 100%), so we shouldn’t do it at all, and if we do it, then it shouldn’t be done in an open source, public institution fashion.

This, as the kids say, triggered me, and lead me to summarize my big problem with the Eliezer-Bostromiam AGI doomsday cult once and for all, so I can just link people to this post.

It has two sections, part one about some background to the state of the discussion, part two why AGI isn’t going to happen with the current LLM approaches and people need to stop overhyping already very transformative tech and part three which goes onto why even if there was a chance of us gaining AGI this way, we should still sign the LAION letter and push for public, open source development and wide dissemination.

Note as of 2023-04-02: Might contain typos, require tightened prose and more sources, and might also be unfair with some people I’m frustrated with. I plan to revisit this text. Bear with me.

Edit 2023-04-02 13:28: fixing some typos and specifying some claims.

A short summary of AGI risk history

AGI (artificial general intelligence), like the word AI or the notion of intelligence is fuzzily defined ², but in the AGI risk community generally means one of the following

1. a computer system with close to, or equal to human level capabilities on all or most cognitive tasks
2. a computer system with better than human level capabilities on all tasks
3. AI Satan, destroyer of worlds, creator of paper clips

where the last line is not meant as a dismissive insult, but an actual description of small parts of the community which mainly engage in fear based, emotional reasoning and catastrophizing with religious overtones rather than sober thought about the implications of tech. However, I need to emphasize, this is the minority. Most people, when pushed, worry a bit about outcome 2 or 3, but generally worry about the risks posed by 1, and that’s a good enough thing to do.

However, some of the loudest and oldest voices in that conversation, namely the lineage of Nick Bostrom and Eliezer Yudkowsky has spent the last 10ish years pumping out propaganda focused on the horrible doom that could befall humanity in case 2 and 3, funded generously by billionaire endowments out of silicon valley. It’s important to note, that neither Bostrom nor Yudkowsky have ever worked with or created ML systems or researched optimization ³, which matters because while on a high level, you can think about the fixed points of a utility function of a system and assume it will learn it to optimality,but in practice, the parametrization and training methods matter and determine whether you can actually reach that optimum⁴. Or put differently, they make the classic mistake of taking a possible configuration of the world based on first principles and acting as if it is a possible configuration in our current trajectory.

The difference between these two things is that the laws of physics and social dynamics allow for all types of configurations, but whether you can actually reach them in the current instance of the world is a different question - or shorter, I can imagine many plausible things, but that doesn’t make the realistic.

However, Bostroms whole stick is (paraphrasing his own worlds) imagining the worst possible outcome and trying to prevent it, which honestly? I’m glad someone does that.

However, the problem with AGI risk and existential risk in general is that risk is personal: climate change is an existential risk for hundreds of millions of people, even in best case scenario. The species might survive, but for these people, that’s no solace - they aren’t so much facing existential risk as existential doom, probability 1 if nothing radical changes.

And at the same time, billionaires like Musk only care about climate change to get tax credits for overpriced badly manufactured “luxury” cars - they have enough resources to relocate to a new private Island if required - their risk of climate change is only nonzero because of flow-through effects like nuclear war becoming more likely, or the collapse of the economy.

So for the general public and even millionaires, existential risks are things like climate change, small scale nuclear war, authoritarian lock-in, pandemics like COVID where 1-10% of humanity might die etc.

Instead, for billionaires, existential risks are things like AGI, large scale nuclear war, pandemics that wipe out 60-100% humanity, global wealth taxes and an unconditional basic income. Everything else they will either be able to mitigate (see the ultra rich preppers here) or they will be the ones benefiting - authoritarian lock-in is from the powerful to the weak, so it’s not even a risk if you think you’ll be on the authoritarian side instead of the locked-in side.

So Bostrom and Yudkowsky spend a lot of words talking about the doom of runaway computer AI and (in Bostroms case) come up with “solutions” like pan-optica implemented via necklaces which will stop you from leaking dangerous information, but discussions of the use of existing algorithmic systems to influence elections, stop labor from organizing or even just the eerie similarity of large multinational corporations and the shareholder system to a profit maximizing runaway optimizer are conspicously absent in their writing - I’m sure it has nothing to do with where their funding comes from.

Note, I’m not accusing anyone here of being a mustache twirling villain, I’m sure everyone involved is a nice human beings⁵ doing what seems right to them - but the whole point of cognitive biases is that they affect everyone. So risk-salience being different is not something you can smart your way out of, you need to have plurality of decision making (i.e., democracy). Self preservation and status quo bias is strong, so if Bostrom and Yudkowsky get funding for one aspect of their concerns, they’ll focus on that and stop doing the part they can’t change and which will dry up their funding - e.g. pointing out that corporations and states are suspiciously like optimizers, especially with capital being able to buy lobbying. And finally, humans soak up a deep narrative pattern of an apocalypse, which is why we have had a bunch of end of the world scares, partially motivated by technostress. If the world is scary, uncertain and you don’t know how things will go, boiling things down to a binary helps cope with that complexity. The end of the world can be a very comforting idea.

Given this, the idea of a big, end of the world scenario is a very good way to collect funding and a dangerous cognitive trap to fall into. As I said, I unironically think it’s a good thing a professional paranoiac like Bostrom exists, but if mixed with billionares money and a fan-of-AI like Yudkowsky, the mix is an AGI doomsday cult, where we need to find the right incantation to seal Satan away before all is too late - and in the histeria this induces, the question of what happens if AGI isn’t the end of the world is assumed to be a utopia, since there is only the binary - we find the incantation there will be utopia, otherwise extinction.

In a mode like this, there is no room for AGI or AI being easy to control, but that control being abused, or questions of politics and balance of power between stakeholders - there is only “humanity” and “the AI”.

The question

What would you do if the end of the world doesn’t come?

is completely neglected in these analyses as all being the “good” scenario.

Now, some good definitely came from Yudkowsky, Bostrom, Stuart Russel, FHI/FLI etc. engaging with the public on this, and I know people funded from the same money troughs who are doing lovely work.

But this elitist perspective on AGI risk has severely poisoned the well and only now with the EFF, Mozilla and other activist groups getting similar funding we see more sober AI risk being discussed which - surprise - is mainly of the form “powerful entity or state automates oppression of poor people”, as in the Rotterdam AI report by lighthouse reports.

Carla Cremer pointed out this problem on how we define risks two years ago and has continued to speak out about this, and I hope that the discourse will shift towards questions about legitimacy of control, not just control per se.

Why LLMs are not, and will not become, AGI

Now, let’s say you buy into the idea of AGI being risky, are LLMs likely to get us there? Or in general, is fast takeoff (the AI going FOOM singularity style) a likelihood?

The answer, in brief, is no.

The last 10 years of AI progress has been a mix of maybe 10% or 20% percent algorithmic improvements and 80% increase in compute and us getting better and more creative about shoveling more data into ever larger models. This is not nothing and it’s incredibly useful, but the debate over AGI requires us to ask whether this approach can get us there, and the answer, most likely is no, for the following reasons:

1. We are likely to run out of training data in form of text and images soon, next up are audio and video, but then that is kinda it - no more nicely structured input to memorize and autocomplete…
2. and that is all these language models do, as evidences by e.g. the glitch token line of investigations, the horrible performance of GPT3.5 on 20 questions and - well, any critical interaction with the system. They memorize the training set as a set of feature-chunks (so not a 1:1 copy but useful snippets) and then regurgitate them in an associative fashion. That is different from reasoning, from planning, or even from how humans use language which is tied to a self loop…
3. which the transformer architecture inherently cannot express: it is a learned, \(8k\)-token sliding window filter giving you a prediction of the next token given the input - nothing more, nothing less. If you put in a lot of effort, you can steer this to be useful by giving inputs you know will be likely to yield useful autocompletes, but that’s like setting up a bunch of dominoes so they will spell out “Fish” when tipped over - if you claim the dominoes or gravity wrote the word, you are misattributing⁶
4. In order to develop a self loop, you need state, which is the one thing the transformer architecture tried to do away with because you need to be stateless for parallelized training
5. even if you had the capability for a state loop, you would need to train the model in a way which induces causal reasoning, something which I am increasingly convinced requires an inductive bias towards a “self”. This is simply because having a notion of self makes it easier to learn which part of the intervention was you and which part was the causal consequence
6. but even if you put all of this into an ML system, you then need to train it in an online fashion - i.e. one step at a time, as you are doing things. You can’t easily parallelize interactions with the real world, you need to keep track of the off-policy bias and training becomes unstable easily - and the whole apparatus LLMs relies on large batch, very careful training in order to not become unstable
7. finally, even if you handled all of this, the sample complexity of learning things in nonstationary environment via reinforcement learning scales very unfavorably⁷

Now, FOOM-AGI riskers will swat away all of these with “but that’s no proof that it’s impossible”, to which I answer “you have no proof that I am not god the almighty either”.

This is a community which has spent 10 years rationalizing away the core logical fallacies of their idea construct namely

• god of the gaps aka argument from ignorance, i.e. “you can’t prove AGI is impossible or safe, therefore it’s inevitable and unsafe”⁸
• Pascals mugging a.k.a. “don’t use probability theory in an open world setting”. If you try to reason about something using probabilities with no grounding of it’s impact nor it’s probability, you can just make up whatever impact and probability your predisposed conclusion needs and go with it

The frame of discourse is now one in which the burden of proof is not on them to argue that AGI is actually coming and it’s a big risk, but on you for arguing why we can take the chance, however slim at AI Satan emerging.

That’s why I brought in the AI Satan earlier and why I call this subset a doomsday cult, because it is a belief system which induces a form of conspiratorial reasoning and cognitive dissonance.

They keep predicting AI will improve and jump, and any advance is taken at face value if it is in favor of that narrative, while counter-examples are reframed as actually proving their point or not mattering because of some vague “future advancement”.

Examples of this include:

• the aforementioned glitch tokens (which are just a funny detail of training, not something about what the models actually learn) and 20 questions (which is relatively new, but just the next challenge to overcome)
• adversarial examples, which uncovered that neural models learn very simplistic and brittle models, as theorized previously but difficult to prove? Taken as examples of how little we understand about neural networks and dangerous unexpected scaling
• alphafold/alphacode: the former can be described as a human made model of protein folding with neural networks in strategic places to learn heuristics from data - but the discourse describes it as learning protein folding without human supervision. The latter is a human constructed system to solve programming puzzles via an LLM - which it does by generating millions of attempts, filtering it to 100 attempts, trying them and calling it “solved” if one succeeds, yielding something like 30% success rate on the test set. This is even less impressive if you know that competitive programming is basically “recognize which algorithm you need to implement and do it quickly”, not a real programming task where you need to break down requirements and design a system de novo.
• the line of work on mis-generalization in RL: it has been long known that if you do not randomize your environment and regularize your RL agents, they will learn shortcut solutions to a given task. The papers in question do not do this, and then describe the failure to learn the task as given as “mis-generalization”. There is no generalisation here, the thing just failed to learn your task.
• language models can be goaded into competing sequences according to a pattern you give to them, e.g. if you count “1 2 fish 4 5 fish 7 8 fish” it has a decent chance of continuing the sequence “10 11 fish” etc.
• This makes sense, since they are trained to sample from \(p(x_{t+1}|x_{t},\dots,x_1)\). However, this has now been termed “in context learning” (a redefinition of the learning term since no parameters are updated) and the community is now similarly discussing “simulator theory”, “the wa-luigi effect” and a whole bunch of other new-speak. All because they are set on these models being more powerful than just a language autocomplete with some text snippets memorized. The (actually very nice!) insight “polite people are always polite, rude people are sometimes polite” could used to frame things in terms of conditional probabilities - e.g. thinking on how to frame prompts such that in your dataset the prompt would come from reasonable discourse.
• Instead they talk about it in terms of “summoning the simulacra” of an evil persona - which is an incredibly dramatic way to express “the model has associates input which is sometimes rude as having high probability of rude output”.

It’s a generation of AI enthusiasts which has self-indoctrinated themselves to look for the AGI in systems which, taken soberly do not exhibit it and this sloppy discourse has now started to affect even researchers I respect immensely ⁹.

It is impossible to train an algorithm to solve NP-hard problems with polynomial samples. Four month old Infants outperform ML systems at generalization While you might be able to train an AGI some day and we will have more and more advanced systems which will be incredibly useful, we do not have a roadmap to AGI.

And I make a testable prediction that anyone who tells you otherwise in the current discourse is either

• working for a company benefiting from AI hype
• pushing their own research agenda on capabilities or safety
• is trying to lobby for regulatory capture/exclusion of the public to source code, data and models

where the last point breaks down into a) well meaning people without an understanding of AGI trying to be extra cautious for their risks (emotional reasoning) and b) malicious actors who want to monopolize power or c) people who got tech-socialized in Bostroms and Yudkowskys horror universe of paperclip optimizers and Voldemort-super intelligence.

Why AI should be publicly controlled, open source and widely spread even if it were likely and even if we were close

TL;DR; intelligence alone doesn’t buy you much, balance of power, security through obscurity does not work and we did all of this during the crypto wars already.

Now, with this background, let’s get back to LAION and the CERN of AI.

Keep in mind we are already in a world where OpenAI, Conjecture etc. exist, all for profit companies¹⁰. What also exists: Pangu, a trillion parameter Chinese state LLM and Russia, currently rogue state invading Ukraine and a whole slew of other proprietary tech companies using AI systems for state and corporate control.

Who would argue against the existence of a public institution that freely shares models, data and know-how in an effort to commodify and really democratize¹¹ AI access? Notably, an institution that calls for AI safety to be a first principle and which would actually develop open AI so people can actually check and not rely on “trust us bros, we asked our friends at this other Nonprofit and they say it’s safe”. Who would do that?

Well, I already started this essay with an answer, but just to give you my assessment summarized

• actors who stand to make billions of dollars from artificially monopolizing the tech and want to do regulatory capture
• AGI existential riskers who
• Ayn Rand like “the state should only protect the property of those who have it, not any other rights” anti-democracy dudes (it is mainly dudes for some reason).

AI, AGI, whatever you want to call it will affect the balance of power, most importantly that between labor and capital. In the absence of UBI and strong political commitments to democracy, the only avenue we have is to commodify AI models as quickly as possible, to minimize the comparative advantage between small companies and individuals and billionaires with hundred of GPUs. Otherwise, those already in power¹² hoover up the efficiency gains of this new tech, except that unions won’t have any leverage any more to distribute the wages like they did previously and they will have staggeringly more powerful surveillance and protest busting capabilities.

So that’s reason one why we need to have public access to the ability to develop, train and deploy advanced AI and AGI systems: otherwise, we destabilize the balance of power in a way which enables authoritarian lock-in.

Reason two is closely linked to this, and that is that inAI/AGI alone doesn’t buy you much. All the threat-models I am aware of rely on you having access to some other resources, which are already the bottleneck and will become so even more with AGI around

• want to influence the masses? You need a platform, not content, content is already cheap
• want to phish people? You need insider info to do spear phishing and/or an anonymous email and cash extraction system
• want to manipulate stocks? You need capital, pass KYC checks etc
• want to hack a power grid? You probably need physical access

Maciej Cegłowski coined this last one the “argument from hawkings cat” which sadly isn’t respected as an argument by AGI riskers, partially because it doesn’t respect the faux-serious tone adopted by people in a community which unironically uses “the waluigi effect” as a term…

That is to say, I consider the argument valid and for some reason, nerds like me who derive a good chunk of our self worth from what is called “intelligence” tend to overestimate how much power “intelligence” gives you¹³

Reason three is that without public access to the models, we can’t find the failure modes and we will go back to redlining. The lighthouse reports behind the scenes notes that without access to the model (kudos to Rotterdam!) they couldn’t have uncovered what they did - and no actor is as motivated to find failures in their models as those affected by them. If you are not the one being acted on, you can mitigate risk by shifting it, hiding it, lobbying against having to pay damages…everything we see right now with the oil industry and the climate crisis. Privatize gains, democratize externalities.

The only way to make high quality and safe models a priority is to have the ability to probe them for failures and make them public - something that we know in software for decades now, if you allow actors to engage in [security theater](https://en.wikipedia.org/wiki/Security_theater] they will, because it’s cheaper.

The final reason is the same line of arguments as from the Crypto wars, where powerful actors try to keep a monopoly on strong crypto because “bad people” (terrorists, communists, capitalists, you know “the enemy”) might use them.

The reply to this, and similar arguments in the ML world is: these people will get access anyway, and then they will encounter systems with not defenses against them, and be at an advantage vs. the common good person.

If you try to stop terrorist from creating neurotoxins by denying them optimization model to do so, then your failure mode is them getting the model, using the supply chain and the publics’ gullibility and unawareness of this risk to get the ingredients and creating them.

If you instead have every high school teacher show how easy it is to get the formula from one of these models to drive home the point of why there is so much bureaucracy around neurotoxin ingredients and why it’s important follow it and report deviances, your failure model is a whole system of caution malfunctioning…and you have a whole team of white hats developing counter-toxins, maybe even making whole classes of toxins obsolete (examples from the software world are modern defenses against SQL injection or buffer overflows).

Having only states, the rich and the bad guys who side step states have access to powerful tools robs us of our ability to defend ourselves and maintain a democracy in which the people are truly sovereign.¹⁴

Even if we are talking about AI and AGI tools. And that’s why you should sign LAIONs petition.