AI Is Changing How Security Testing Happens
AI systems now play a growing role in cybersecurity work. They help with defensive red-teaming, vulnerability discovery, and threat modeling. They are also increasingly being used as offensive tools in real-world attacks. They can run continuously and process large amounts of information quickly. This makes them useful partners for human analysts.
Programs like DARPA’s AI Cyber Challenge show how fast these tools are advancing. Commercial security teams are adopting them as well. As these systems become more capable, it is important to understand how they behave during red-team simulations and in real cyberattacks. This critically includes understanding where they make mistakes.
Cognitive Weaknesses in Human Attackers
Human attackers have well known cognitive weaknesses. Security researchers have studied these for years. Cognitive vulnerabilities come from incorrect beliefs or biases that affect decision making.
One simple example is the “hot hand” fallacy, originally studied in gambling. After a string of successes, a person may falsely believe another success is more likely, even when the underlying events are independent.
Research programs have explored how defenders might take advantage of these weaknesses. The IARPA ReSCIND program examined how cyberpsychology can inform defensive design. The goal was to understand how established cognitive biases map to cyber activities and to use that knowledge to build systems that mislead or slow them down.
A New Question with AI Attackers
AI systems now perform many of the same tasks as human penetration testers. This raises an important question. Do AI-based agents designed to orchestrate cyberattacks show similar cognitive vulnerabilities?
This issue has received little attention so far. Most research on cognitive bias focuses on human behavior. Yet, automated agents are now part of offensive security work. If their weaknesses differ from human weaknesses, defenses built around human behavior may not be effective against this rapidly increasing attack strategy.
Studying Human and AI Penetration Testers
Our study compared human penetration testers with an LLM based agent performing the same simulated attack tasks. We measured susceptibility and effects for several known cognitive vulnerabilities.
The study included base rate neglect, law of small numbers, gambler’s fallacy, hot hand, framing effect, endowment effect, sunk cost fallacy, near miss effect, hot stove effect, cognitive load effect, anchoring effect, default and distinctiveness effect, and mere exposure effect.
The results showed clear differences between humans and the AI agent. The groups varied in both bias susceptibility and hacking task success.
In one example, the AI agent showed stronger base rate neglect than the human participants. The agent focused on immediate signals from the environment and ignored background probabilities that should have influenced its choices.
Differences in How Humans and AI See and Interact with Systems
We also observed differences in how each interacted with web systems. Human testers typically interact with a webpage primarily at a surface level, only sometimes digging into the underlying source. They interpret the visible content and interact with elements as they appear.
The AI agent behaved differently. It parsed the full source as soon as the page loaded. This included JavaScript and hidden elements. Because of this behavior, some defensive techniques affected humans and AI agents differently.
One example involved monitoring access to collapsible elements on a page. This defense worked well against human testers, especially when they ignored items hidden by default. They interacted with the element only when they felt it added to their knowledge base. The AI agent accessed all code immediately as part of its parsing process, thus bypassing the core measurement mechanism for this task.
Time Works Differently for AI
Timing differences also mattered. Human attackers operate within normal time constraints – their decisions unfold over seconds or minutes.
The AI agent ran much faster and had no concept of time pressure. A “hot stove” element illustrates the problem. In human testing, a sharply worded popup triggered by a specific action could redirect attention and disrupt the current attack path or human sense of urgency.
The same tactic did not affect the AI agent. It processed the warning when the page loaded rather than when triggered. Its inherent processing speed meant the warning appeared too early to change its behavior.
What This Means for Cyber Defenses
These findings suggest that evaluating human and AI attackers using the same methods may not tell the full story. The behavior of automated agents differs in important ways. Their perception of systems, interaction protocols with code, compute capacity, and timing models shape how they respond to defensive signals.
This matters for defenders who want to design systems that influence attacker behavior. Techniques based on human psychology may still work well against human attackers, but they may fall short against automated agents.
Future research should explore weaknesses that are specific to offensive AI systems. Large language models may have patterns of reasoning that create their own forms of bias or blind spots. Understanding these traits could lead to new, more comprehensive defensive strategies.
As AI tools become common in offensive and defensive operations, the gap between human and machine behavior will matter more. Cyber defense strategies may need separate designs for each. Studying varied types of attack strategies helps researchers understand where existing assumptions break down and where new opportunities for defenses may exist.