New ConfusedPilot attack targets AI systems with data poisoning

University of Texas researchers at the SPARK Lab in Austin have identified a novel cyberattack method called ConfusedPilot that targets Retrieval-Augmented Generation (RAG)-based AI systems such as Microsoft 365 Copilot.

The team, led by Professor Mohit Tiwari, CEO of Symmetry Systems, discovered how attackers can manipulate AI-generated responses by inserting malicious content into documents referenced by the AI.

This could lead to misinformation and poor decisions across organizations.

With 65% of Fortune 500 companies adopting or planning to implement RAG-based systems, the potential for widespread disruption is significant.

The ConfusedPilot attack method requires only basic access to a target’s environment and can persist even after the malicious content is removed.

The researchers also showed that the attack could bypass existing AI security measures, raising concerns across industries.

This is how ConfusedPilot works

• Poisoning the data environment: An attacker adds specially crafted content to the documents indexed by the AI ​​system
• Document retrieval: When querying, the AI ​​points to the damaged document
• AI misinterpretation: Using the malicious content as instructions, the AI ​​may disregard legitimate information, generate misinformation, or incorrectly attribute its response to credible sources
• Resistance: Even after removing the malicious document, the corrupted information may remain in the system

The attack is particularly concerning for large companies using RAG-based AI systems, which often rely on multiple user data sources.

This increases the risk of attacks as the AI ​​can be manipulated using seemingly innocuous documents added by insiders or external partners.

“One of the biggest risks business leaders face is making decisions based on inaccurate, draft or incomplete data, which can lead to missed opportunities, lost revenue and reputational damage,” said Stephen Kowski, Field CTO at SlashNext.

“The ConfusedPilot attack highlights this risk by showing how RAG systems can be manipulated through malicious or misleading content in documents that were not originally submitted to the RAG system, resulting in AI-generated responses being compromised .”

Read more about AI security in companies: Tech experts point to critical skills gaps in AI security

Mitigation strategies

To defend against ConfusedPilot, the researchers recommend:

• Data access controls: Limiting who can upload or modify documents referenced by AI systems
• Data audits: Regular checks to ensure the integrity of stored data
• Data segmentation: Isolate sensitive information to prevent the spread of compromised data
• AI security tools: Using tools that monitor AI output for anomalies
• Human supervision: Ensuring human review of AI-generated content before critical decisions are made

“To successfully integrate AI-powered security tools and automation, organizations should first evaluate the effectiveness of these tools in their specific context,” said Amit Zimerman, co-founder and chief product officer at Oasis Security.

“Instead of being swayed by marketing claims, teams need to test tools against real data to ensure they deliver actionable insights and uncover previously unseen threats.”