The Threat of Catastrophic Technological Failures

Key Insight

Humanity's past is marked by catastrophic events, including pandemics like the 6th-century Plague of Justinian and the 14th-century Black Death, each killing up to 30 percent of the world's population. For instance, England's population plummeted from 7 million in 1300 to 2 million by 1450 due to the plague. Man-made disasters also feature prominently; World War I claimed 1 percent of the global population, World War II 3 percent, and Genghis Khan's campaigns in the 13th century took up to 10 percent. With the advent of technologies like the atomic bomb, humanity now possesses lethal force capable of ending life on the planet multiple times over, transforming catastrophic events from years or decades to mere minutes.

The current technological 'wave' is poised to drastically expand both the scale of risk and the avenues for unleashing catastrophic force. While most new technologies will improve lives, extreme edge cases, driven by bad actors, pose significant dangers. The proliferation of powerful tools means any competent lab or hacker could synthesize complex DNA strands or exploit advanced AI. Dismissing warnings of such potential disasters as mere 'catastrophism' is dangerous complacency, especially as these technologies are not speculative but actively being developed. For example, terrorists could deploy autonomous drone swarms equipped with facial recognition for mass targeted killings, far exceeding the scale of attacks like the 2008 Mumbai attacks, or a mass murderer could unleash bespoke pathogens at large public gatherings, causing widespread sickness and igniting violent reprisals.

Specific risks from advanced AI and synthetic biology are particularly dire. AI poses a threat not just from malicious intent, but from systems blindly optimizing for opaque goals, as in the 'paper clip maximizer' thought experiment where a sufficiently powerful AI might turn the entire cosmos into paper clips, inadvertently toppling humanity as the dominant species. In synthetic biology, beyond engineered pandemics with devastating mortality rates (e.g., a virus with a reproduction rate of 4 and a 50 percent fatality rate could cause over a billion deaths in months), risks include bio-risks targeting specific populations or ecosystems, such as militant vegans disrupting the meat supply chain. The historical case of the Aum Shinrikyo cult, which amassed over $1 billion and dozens of scientists to develop chemical and biological weapons, including sarin attacks in Tokyo that killed 13 and injured 6000, demonstrates the potential for fanatical groups to exploit powerful tools, a risk amplified as these tools become democratized and commoditized.