The modern world is incredibly insecure along a wide variety of dimensions - because it’s not a problem. Usually. No-one is trying to exploit the security of your email server, most of the time, so it’s fine if it is unpatched. No-one is trying to hack the internet alarm clock radio you have on the counter, or the toy drone your kid is playing with, or even your car’s radio and (for non-celebrities, at least,) your twitter or facebook login, so they can be insecure. And in general, they are. Even things you might worry about -your bank account, or your pacemaker - are generally pretty insecure. This matters much less when the cost of hacking is relatively high, and there are richer and/or easier targets than yourself.
Subscribe
The FBI releases annual data on “internet crime losses,” and the multi-year trend has been around a 40% growth in losses each year, currently approaching $20b in reported and direct losses, though only a small fraction are related to hacking, the rest are non-automated scams and cryptocurrency-based losses (often also due to scams.) This is worrying, especially when you realize that indirect damages is not included and not everything is reported - but it is not catastrophic, unless the risk changes.
The Case for Concern
The rise of low-cost LLMs is changing the bar for how hacking can be done, and how scams can be automated, each changing the cost-benefit calculation rapidly. We are already starting to see that along with the increasing capabilities of frontier LLMs, hacking is being automated. The AI Risk Explorer compiles events, and shows that sophistication has been rising rapidly - the 3rd quarter of 2025 saw the first AI-powered ransomware, and the first agentic AI powered attacks. Over time LLMs run by nefarious groups will increasingly be capable of automated recon, vulnerability chaining, password spraying, SIM-swap orchestration for technical multifactor authentication, social attacks on multifactor authentication, and phishing kit generation - not to mention C2 automation for botnets and superhuman speed response to countermeasures. None of this will surprise researchers who have paid attention - and recent news from Anthropic makes the advances clear. But along with sophisticated attackers, it’s likely we’ll experience automated exploits of previously secure-by-apathy targets at scale. And the scale of the vulnerability is shockingly broad.
Of course, LLMs will help with cyber defense as well. But even if the offense-defense balance from AI favors defense, that won’t matter in the short term! As Bruce Schneier pointed out, the red team will take the lead. And while AI isn’t able to do this reliably yet, even a couple percentage point success rate would easily overwhelm response capabilities. This is especially true with various classes of scam, where the success rate is tiny but it is still economically viable, and that rate could go up tremendously with high volume personally tailored scams. And model developers try to monitor and stop misuse, but the threats don’t match the research evaluating the risks, and they likely only find a fraction of what is occurring.
So - how worrying is having your fridge hacked? Probably not so worrying - unless it’s inside your home network, and can be used to infect other systems directly, allowing hackers into your sensitive systems. How worrying is it to fall for a scam email? Not so bad, as long as you don’t share passwords across accounts - which you should not do, and you should use 2-factor authentication. But people who are using texts or authenticator apps on vulnerable phones could still be attacked from inside the home network, if and when sophistication of attacks rises.
In 2024, LLMs started to automate a variety of scams and “social” attacks, like convincing less sophisticated users to give away their banking passwords, or send them cryptocurrency. This is somewhere on the border between a social vulnerability to persuasion and cybersecurity threat, but is enabled by insecure digital infrastructure - perhaps starting with long-understood issues with insecure and non-verified phone communications infrastructure that persist, the increasing availability of unlabeled AI outputs, despite methods to fix this, and other issues.
The Bigger Targets
And so far, we’ve only discussed individuals. The risks from near-term advances also include any organization that doesn’t secure itself from the compromise of their employees, as well as organizations with any internet-exposed surface, including sensitive agencies. These organizations regularly have unpatched systems and insecure user authentication for their employees.
Of course, many of the more important of the vulnerable targets will be able to detect even novel exploits, or address insider threats, and respond effectively, as occurs today. And when humans are doing the exploits, these responses might be fast enough to minimize damage. But once we have frontier LLMs that can run dozens, if not hundreds or thousands of parallel instances to exploit the systems, and run the command and control, and quickly respond to anything done at human speed, it seems plausible that for some classes of incidents, rapid isolation may be the only reliable stopgap - and even that might happen too late to prevent key data from being exfiltrated, or prevent the hacks from expanding their exploits to elsewhere inside the networks.
Should This Be A Priority?
There are certainly other larger risks, including misuse by terrorists, potential future bioengineering of novel pathogens, everyone dying, or other posited future risks. Each of these is more devastating than a cyber attack, if successful. But these are areas where tremendous efforts in detecting and mitigating threats are already occurring. Terrorism has been a focus of successful (if often overzealous) defensive and preventative efforts for decades, and increased threats are worrying but not obviously transformative. Similarly, Governance options for biosecurity are feasible, and between those and traditional public health for fighting diseases, there’s reason for optimism that we will succeed in putting in place sufficient safeguards.
In contrast, cybersecurity’s playing field has long put defenders at a technical disadvantage, and arguably an organizational one as well. As Caleb Withers at CNAS pointed out recently, the scales could easily tip further. That means that unless major changes are put in place, AI progress will lead to increasing vulnerability of individuals, and easier and more routinely devastating attacks against organizations. This isn’t the existential risk we’ve worried about; it’s far less severe, but also very likely more immediate. Models might not be there yet, though this is unclear given that the linked analysis isn’t inclusive of GPT-5 or Claude 4, and the recent (exaggerated) Anthropic report of an “autonomous” (e.g. partially automated) hack by China is certainly not demonstrating that unsophisticated actors could mounter larger attacks at scale, but we’re getting there quickly.
Addressing the Threat
Despite the challenge, we know how to address many parts of this problem, and have for quite some time - though many of the critical approaches are unrelated to AI. The lack of robustly verified or at least audited and secure code for operating systems and other critical cyber infrastructure is striking - we’ve known about them for decades, but we’ve seen no serious effort to eliminate them. The vulnerability of our phone and other communication systems to fraudulent misuse is another glaring gap - and while technical efforts are progressing, albeit far too slowly, the fundamental gap is that international cooperation is not occurring, and untrusted networks remain in place.
On the AI front, the lack of even token efforts to reliably tag AI and other digital outputs so they can be distinguished from humans shows the issue isn’t being taken seriously (again, despite methods to fix this.) And there’s also no concerted effort to prioritize cyber-defense applications of AI, nor to hold AI companies legally responsible if their systems are used for nefarious purposes.
For all of these, the first step is to notice we all have a set of serious problems in the near future, and little is being done. In cases like this, the problem will scale quickly, and prevention is far cheaper than response. Unfortunately, the companies capable of solving the problems are not responsible for doing so. There’s unfortunately little reason for the model creators to care, especially given how lukewarm they are about current misuse, and every reason to think the current widespread cybersecurity apathy will continue long past the point where the costs imposed by scalable attacks have become unacceptably high.
Thank you to Asher Brass for an initial conversation about the topic, and feedback on an earlier version of the draft.
