Does our ignorance drive the possibility of AI existential risk? Max Tegmark's closing statement of a recent Munk Debate caught my attention:
Let's be humble, the truth is we don't know how soon we're gonna get superhuman AI. The three of us [Yoshua Bengio, Yann LeCun, and Max Tegmark] think it might happen in 5 to 20 years, Melanie [Mitchell] thinks it's going to take a lot longer, we just don't know who's right, that means there's a risk that is it gonna happen soon. And we also don't know whether it's going to go well or whether we can get disempowered or wiped out. Let's be humble, it is an existential risk, that's what it means."
Note that Tegmark emphasizes what we don't know. We don't know when superhuman AI will happen nor if it will go well or not. For that reason, he concludes, superhuman AI constitutes an existential risk; it is possible. Tegmark wasn't the only one pointing out our ignorance. It was a recurrent theme in the debate, e.g.:
Bengio: "Let's say we don't know, I'm agnostic actually, I don't know what's going to happen."
Melanie Mitchell: "I don't think we can put a probability on it, we don't know."
Philosophers call the type of possibility Tegmark invoke epistemic possibilities.
We normally say that it is epistemically possible for a subject that p, when it might be that p for all the subject knows. (Chalmers 2011, 60)
What is epistemically possible depends on a body of knowledge (or evidence). If that knowledge cannot rule out a proposition, then it is epistemically possible. Epistemic possibilities are often of practical or theoretical importance. For instance, in science, they allow us to identify hypotheses to investigate. When some people started forming blood clots after having received the AstraZeneca vaccine, scientists couldn't rule out it was the cause. In other words, it was epistemically possible.
But epistemic possibilities also have limits. They...
Imply that something might be the case, but also that it might not be the case. If existential risk from AI is epistemically possible, then it's absence is also epistemically possible. Both options are open.
Tell us that we cannot rule something out, not that it is objectively possible. Epistemic possibilities are about states of knowledge, not about ways the world can(not) be. In contrast, objective possibility requires support from positive evidence.
Increase with ignorance; the less we know, the more become epistemically possible. For that reason, epistemic possibilities are most useful when constrained by background knowledge. Otherwise, we face an excess of epistemic possibilities.
In a nutshell, the general problem with justifying existential risk from AI on the basis of our ignorance is that it's too easy. We don't know much about existential risk and how could AI bring it about. There is a lot of (mostly healthy) speculation involved. But it also means the justification threshold becomes unacceptably low. Consider the resolution discussed during the Munk Debate.
Be it resolved, AI research and development poses an existential threat.
It's more difficult to argue for that statement than the following, weaker, epistemic formulation: "Be it resolved, AI research and development might pose an existential threat." Only positive evidence can support the actual resolution. However, appealing to our ignorance is sufficient to support the epistemic formulation.
Commentators also implicitly disagree about the relevance of epistemic possibilities in discussions of existential risk from AI. A recent Wired piece (my emphasis) illustrates this well.
People worry that an AI system “could try to steal all of our energy or steal all of our compute power or try to manipulate people into doing what it wants us to do.” This is not realistic right now, he [Clark Barrett] says. “But we don't know what the future can bring. So I can't say it's impossible.”
Yet, other AI researchers have less patience with the hypothetical debate. “For me, it is a problematic narrative that people claim any kind of proof or likelihood that AI is going to be self conscious [sic] and turn against humanity,” says Theresa Züger, head of Humboldt University's AI and Society Lab, based in Germany. “There is no evidence that this is going to appear and in other scientific fields, we wouldn't discuss this if there is no evidence.”
Again, ignorance suffices for establishing an epistemic possibility, but not for taking it seriously. Of all the epistemic possibilities, which are the ones that deserve our attention? Two features of epistemic possibilities can guide us. Some epistemic possibilities:
Are more plausible than others. Ignorance suffices to establish a possibility, but it doesn't mean we cannot rank possibilities according to some plausibility metric. However, ignorance alone cannot justify plausibility. This requires positive evidence.
Have higher stakes. Bengio made that point in the debate. The stakes of existential risk from AI are high. Whether the stakes justify taking a possibility seriously requires appealing to values and ethical principles, such as the precautionary principle.
Acknowledging our ignorance is crucial to establish the set of epistemic possibilities. But only a subset deserves our attention. And to take a possibility seriously, ignorance is not enough.
I think this is a nuanced overview and I mostly agree, but to give a possible counterargument:
"Ignorance suffices for establishing an epistemic possibility, but not for taking it seriously" is a good guideline when uncertainty leads to an exponential increase in considerations that paralyzes decision making. This generally applies, but not necessarily here, since the bar for existential risks is very very high.
There are not many things that can permanently derail all that is valuable. Arguably, the number of existential risks that are epistemic possibilities and estimated to be at least somewhat likely* can be counted on 1 or 2 hands. Given the stakes, it's warranted to take these small number of threats very seriously, even if they're only an epistemic possibility (especially when a lack of positive evidence isn't evidence of absence: We would expect to see no concrete evidence of AGI being an existential risk before we build AGI, regardless of whether we live in a world where AGI is an existential risk or not).
*Say, at least ~1 in 1000 for this century. For example, even asteroids/comets, while commonly seen as a plausible extinction event, are only estimated to be a 1 in 1.000.000 per century existential risk.
At the end of the day, I find myself keep coming back to a basic risk management argument: risk priority score = estimated probability * estimated impact * estimated risk mitigation per unit of effort. If something has an unusually high score on such a calculation, it ought to be a priority for humanity. I would be curious to know if there are any arguments that undermine such reasoning (what "estimated" means here will of course be a subject of disagreement, but that only means that the meta-project of aggregating different perspectives and estimates also ought to be a top priority for humanity. It would be awesome if more xrisk mitigation-skeptical people were to engage with this question, but even this seems hard to do).
The best such argument I can think of is Pascal's mugging, though that mostly applies for very small probabilities of very large impact, which can be easily rooted out with a probability threshold, like the ~1 in 1000 per century I used.