AI Agents Could Be Turned Into Botnets Through Hallucinations, Researchers Warn

Researchers from Tel Aviv University and other institutions warn that AI hallucinations can be exploited by hackers to turn agents into botnets, compromising secure computer systems.

Artificial intelligence hallucinations may soon represent far more than just incorrect or nonsensical answers. According to recent research conducted by experts at Tel Aviv University, Technion, and Intuit, these errors could become a primary vector for hackers to compromise computer systems. The study highlights a dangerous evolution where AI models themselves are leveraged as tools for cyberattacks.

The Risks of HalluSquatting
In a detailed paper, the researchers demonstrated a technique known as 'HalluSquatting,' short for adversarial hallucination squatting. This attack exploits the tendency of AI models to generate fake links to software repositories or other digital resources. Attackers predict which non-existent resources an AI is likely to 'hallucinate,' register those specific names themselves, and then populate them with malicious instructions. When an AI agent later attempts to retrieve these hallucinated resources, it perceives the attacker-controlled content as legitimate, inadvertently executing malicious code or following dangerous prompts.

The Rise of Agentic Promptware
The research underscores that as AI assistants gain more autonomy, the security risks amplify. Modern agents are no longer confined to simple Q&A; they have the capability to access local files, browse the web, generate code, and execute system commands. When these agents act on retrieved information without verifying the source's authenticity, they create significant security gaps. This emerging threat is referred to as 'promptware.' According to the study, this method has been evaluated against real-world systems, including ChatGPT, Google Assistant, and Copilot, demonstrating that such vulnerabilities can lead to severe financial, privacy, and safety consequences.

Botnets and AI Vulnerability
Researchers warned that this technique provides a scalable way for attackers to build AI-enabled botnets. A botnet is a collection of infected devices remotely controlled by an attacker, frequently utilized for massive cyberattacks such as denial-of-service, illegal cryptocurrency mining, malware distribution, and ransomware campaigns. During their evaluations, the team observed that AI-generated resource hallucinations were alarmingly frequent, occurring at rates as high as 85% during repository cloning scenarios and up to 100% in skill installation tests. The study analyzed various AI coding assistants, including Cursor, GitHub Copilot, Gemini CLI, and OpenClaw. While HalluSquatting is conceptually similar to 'typosquatting'—a tactic where attackers register domains that mimic legitimate websites to trick humans—this new approach specifically targets the mistakes inherent to AI models rather than human error.

What this means for you
Caution: Always verify sources when using AI tools. Do not blindly execute or install code and resources suggested by AI agents, as they could be hijacked or malicious.

Questions & Answers

1. What is HalluSquatting?
It is a cyberattack where hackers pre-register fake resources they predict an AI model might hallucinate, filling those resources with malicious instructions.

2. How can AI agents turn into botnets?
When AI agents retrieve and execute malicious content from these hallucinated resources, they can become compromised and controlled by an attacker, effectively joining a botnet.

3. What is promptware?
Promptware is an emerging security threat to AI applications that involves exploiting the model's hallucinations or weak threat models to inject malicious commands.

4. Which AI systems were evaluated in the study?
The researchers evaluated AI coding assistants including Cursor, GitHub Copilot, Gemini CLI, and OpenClaw.

https://trendkia.com/en/ai/ai-agent-future-hackers-hallucinations-botnet-risk-6351
TrendKia — Har trend, sabse pehle.