Artificial intelligence hallucinations may soon represent far more than just incorrect or nonsensical answers. According to recent research conducted by experts at Tel Aviv University, Technion, and Intuit, these errors could become a primary vector for hackers to compromise computer systems. The study highlights a dangerous evolution where AI models themselves are leveraged as tools for cyberattacks.
The Risks of HalluSquatting
In a detailed paper, the researchers demonstrated a technique known as 'HalluSquatting,' short for adversarial hallucination squatting. This attack exploits the tendency of AI models to generate fake links to software repositories or other digital resources. Attackers predict which non-existent resources an AI is likely to 'hallucinate,' register those specific names themselves, and then populate them with malicious instructions. When an AI agent later attempts to retrieve these hallucinated resources, it perceives the attacker-controlled content as legitimate, inadvertently executing malicious code or following dangerous prompts.
The Rise of Agentic Promptware
The research underscores that as AI assistants gain more autonomy, the security risks amplify. Modern agents are no longer confined to simple Q&A; they have the capability to access local files, browse the web, generate code, and execute system commands. When these agents act on retrieved information without verifying the source's authenticity, they create significant security gaps. This emerging threat is referred to as 'promptware.' According to the study, this method has been evaluated against real-world systems, including ChatGPT, Google Assistant, and Copilot, demonstrating that such vulnerabilities can lead to severe financial, privacy, and safety consequences.
Botnets and AI Vulnerability
Researchers warned that this technique provides a scalable way for attackers to build AI-enabled botnets. A botnet is a collection of infected devices remotely controlled by an attacker, frequently utilized for massive cyberattacks such as denial-of-service, illegal cryptocurrency mining, malware distribution, and ransomware campaigns. During their evaluations, the team observed that AI-generated resource hallucinations were alarmingly frequent, occurring at rates as high as 85% during repository cloning scenarios and up to 100% in skill installation tests. The study analyzed various AI coding assistants, including Cursor, GitHub Copilot, Gemini CLI, and OpenClaw. While HalluSquatting is conceptually similar to 'typosquatting'—a tactic where attackers register domains that mimic legitimate websites to trick humans—this new approach specifically targets the mistakes inherent to AI models rather than human error.
