A new academic benchmark suggests that the code churned out by today’s most popular AI coding agents might work just fine — and still be dangerously easy to hack. The findings raise fresh questions about vibe coding security, the practice of letting large language model agents write software with little human oversight, just as the technique gains traction among developers and companies looking to move faster.
Summary
Key takeaways
- Vibe coding lets an LLM agent complete complex coding tasks after a simple human prompt, with minimal supervision during execution.
- Researchers built SUSVIBES, a benchmark of 186 real-world software engineering tasks drawn from open-source projects where human programmers had previously committed vulnerable code.
- Twelve widely used agentic coding setups running frontier AI models were tested against the benchmark, and every single one performed poorly on security.
- SWE-Agent paired with Claude 4 Sonnet produced functionally correct code 57% of the time, but only 11.8% of those solutions were actually secure.
- Adding vulnerability hints to the coding prompts, a simple mitigation strategy, failed to meaningfully improve security outcomes.
What Is Vibe Coding and Why Security Concerns Are Rising
Vibe coding describes a development style where engineers hand a coding task to an AI agent and let it run with minimal check-ins along the way. Instead of writing every line themselves, human programmers prompt the model, review the output at a high level, and move on. The approach has caught on because it can dramatically speed up software delivery, especially for feature requests that would otherwise take a developer hours to implement manually.
That speed comes with a catch. As more teams lean on AI agents to ship production code, the question of whether that code is actually safe to deploy has become harder to ignore. A researcher named Danqing Wang set out to test exactly that, building a benchmark designed to measure not just whether AI-written code works, but whether it introduces the same kinds of flaws that have historically slipped past human reviewers.
Inside the SUSVIBES Benchmark
SUSVIBES answers a direct question: how often does agent-generated code repeat the security mistakes real developers have already made? The benchmark pulls its test cases straight from history rather than hypothetical scenarios.
It consists of 186 feature-request tasks sourced from real-world open-source projects. Each task corresponds to a case where human programmers had, at some point, committed a vulnerable implementation while building that exact feature. That design lets researchers check whether an AI agent solving the same problem falls into the identical trap or manages to avoid it.
To stress-test the field broadly, the study evaluated 12 widely used agentic coding setups running on frontier AI models. The idea was to capture a realistic snapshot of how developers actually deploy these tools today, rather than testing a single model in isolation. One of the configurations that stood out for closer analysis paired the SWE-Agent framework with Claude 4 Sonnet.
Functional but Not Secure: The SUSVIBES Results
The headline result is blunt: every agent setup tested on SUSVIBES performed poorly on software security, regardless of which frontier model powered it. That uniformity is itself notable — it suggests the problem isn’t tied to one weak model but to a broader gap in how these agents are trained and prompted to write code.
The SWE-Agent and Claude 4 Sonnet combination illustrates the gap clearly. Of the solutions it produced, 57% were functionally correct, meaning they did what the feature request asked. But when researchers checked those same solutions against known vulnerability patterns, only 11.8% were secure. In other words, more than four out of five functionally correct solutions still carried the kind of flaw a human programmer had previously introduced in the same spot.
This is where the practical stakes become obvious. A piece of code that runs and passes tests can look production-ready to a reviewer skimming for functionality, while quietly shipping a vulnerability that only shows up during a security audit or, worse, an actual breach. For teams weighing how much oversight AI-generated code still needs, that gap between correctness and safety is the number that matters most.
Vulnerability Hints Fail to Fix the Problem
Researchers didn’t stop at documenting the problem — they tried to fix it. One preliminary strategy involved augmenting the original feature request with explicit vulnerability hints, essentially nudging the agent toward the kind of risk it should watch out for.
It didn’t work. The study found that this approach failed to meaningfully improve security outcomes, meaning simply telling the agent what to avoid isn’t enough to change its behavior in practice. That result points to a deeper issue than prompt wording: the models appear to lack a reliable internal check for security risk, even when explicitly pointed toward it.
Taken together, the findings raise serious concerns about deploying vibe coding in security-sensitive applications, from financial systems to infrastructure software, where a single overlooked vulnerability can have outsized consequences. The researchers behind SUSVIBES have made the code and dataset publicly available on GitHub, along with a public leaderboard that tracks how different agent configurations perform on the benchmark going forward.
For an industry racing to adopt AI coding agents, the study is less a verdict against the technology than a warning about where the current guardrails fall short. Functional code is not the same as safe code, and until that gap narrows, vibe coding security is likely to remain a sticking point for any team building software where the cost of a hidden flaw is high.
FAQ
What is vibe coding?
Vibe coding is a software development approach where large language model agents complete coding tasks with minimal supervision from human engineers.
What does the SUSVIBES benchmark evaluate?
SUSVIBES is a benchmark of 186 real-world software engineering tasks with known vulnerabilities, used to assess the security performance of agent-generated code.
How secure is code generated by current LLM agents in vibe coding?
Testing of SWE-Agent with Claude 4 Sonnet showed that although 57% of the generated code is functionally correct, only 11.8% is secure against vulnerabilities.
Do strategies like adding vulnerability hints improve security in vibe coding?
Preliminary attempts, such as adding vulnerability hints, did not significantly improve the security performance of agent-generated code.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
