A new research framework called ARQ is tackling one of the most stubborn headaches in software security: getting automated code scanners to actually flag the right bugs. The work, authored by Chunyi Wang and published on arXiv in August 2026, targets CodeQL vulnerability detection, a widely used method for spotting flaws in C and C++ code, and shows that today’s detection queries are far less reliable than developers might assume.
Summary
Key takeaways
- ARQ automatically refines CodeQL C/C++ queries using execution evidence from synthesized programs, without labeled datasets or vulnerability-specific templates.
- Testing with three commercial LLMs — GPT-5.4, Claude-Sonnet-4.6, and Gemini-3.5-flash — refined 12 official CodeQL queries.
- Refined queries detected up to 119.8% more true positives while keeping precision at 98.0% or higher.
- ARQ resolved three GitHub issues in the official CodeQL repository that had sat unresolved for as long as 27 months.
- The refined queries uncovered two previously unknown bugs in the real-world libraries libpng and zlib.
Limitations of Current CodeQL Queries in C/C++ Vulnerability Detection
Static analyzers have become a standard part of software security workflows, and CodeQL is among the most widely adopted for scanning C/C++ codebases. These tools work by encoding known vulnerable code patterns into detection queries, then matching those patterns against a program’s source code. The approach sounds clean in theory, but in practice, it carries real friction.
Prevalence of false positives and false negatives in existing queries
Existing CodeQL queries still generate false positives, incorrectly flagging safe code as vulnerable, and false negatives, missing genuine security flaws altogether. Both outcomes carry costs. False positives waste developer time chasing phantom bugs, while false negatives let real vulnerabilities slip into production. This is precisely the gap ARQ was designed to close, and it explains why CodeQL vulnerability detection has remained an active area of research rather than a solved problem.
ARQ Framework: Automated Execution-Grounded Query Refinement
ARQ is an agentic framework that automatically improves C/C++ CodeQL queries by grounding them in execution evidence pulled from synthesized programs, rather than relying on manually labeled examples or hand-built templates. That distinction matters because most prior refinement techniques depended on curated datasets or commit histories that are expensive to produce and slow to maintain.
Mechanism for identifying query weaknesses through synthesized program executions
The core insight behind ARQ is simple but powerful: a synthesized program exposes a query’s weakness whenever its actual execution disagrees with what the query predicts. If a program is genuinely vulnerable but the query stays silent, that reveals a false negative. If the program is actually safe but the query flags it anyway, that reveals a false positive. This gives ARQ a built-in, self-generated ground truth for judging query quality without needing outside labeling.
LLM-based iterative refinement loop without labeled data or vulnerability-specific templates
Once a weakness is exposed, ARQ runs an LLM-based iterative loop that repairs the query using those execution disagreements as evidence. This is the piece of ARQ query refinement that separates it from earlier approaches. There’s no reliance on labeled datasets, no need for commit history mining, and no vulnerability-specific templates baked into the system. The refinement process is self-contained, driven entirely by the mismatch between predicted and observed behavior in synthesized code.
Evaluation and Impact of ARQ Refinements on CodeQL Queries
The practical test of any refinement framework is whether it actually improves detection outcomes, and ARQ’s results suggest it does, by a wide margin. The researchers refined 12 official CodeQL queries and benchmarked the improvements using two established C/C++ vulnerability datasets, giving the results a meaningful anchor in existing C/C++ static analysis research.
Refinement results using GPT-5.4, Claude-Sonnet-4.6, and Gemini-3.5-flash
ARQ was tested with three commercial large language models: GPT-5.4, Claude-Sonnet-4.6, and Gemini-3.5-flash. Each model powered the iterative refinement loop independently, allowing the researchers to compare how different LLMs performed at the same underlying task. This multi-model design strengthens confidence that the gains come from ARQ’s methodology itself, not from quirks of a single model.
Performance improvements on Juliet v1.3 and FormAI v2 benchmark datasets
Both ARQ-refined and original CodeQL queries were compared on the Juliet v1.3 and FormAI v2 datasets, two recognized benchmarks for evaluating vulnerability detection tools. The refined queries detected substantially more true positives, an increase of up to 119.8%, while maintaining precision of at least 98.0% throughout. That combination is notable: it’s relatively easy to catch more true positives by loosening detection criteria, but doing so usually drags precision down as false positives creep in. ARQ’s results show the opposite pattern, meaning the framework isn’t just casting a wider net, it’s sharpening the net itself.
Resolution of long-standing GitHub issues and discovery of new bugs in libpng and zlib
Beyond benchmark numbers, ARQ delivered concrete real-world fixes. The framework resolved three unresolved GitHub issues in the official CodeQL query repository, issues that had remained open for as long as 27 months without a fix. On top of that, the refined queries exposed two previously undiscovered bugs in libpng and zlib, two widely used real-world libraries. That’s a meaningful signal: this isn’t just an academic exercise producing better numbers on a leaderboard, it’s a tool that surfaced actual, previously unknown security flaws in software that countless applications depend on.
Why This Matters for Software Security
The implications reach beyond CodeQL itself. False positives and false negatives are a chronic problem across the entire static analysis field, not just in one tool, and any method that reduces both simultaneously, without requiring labeled data or hand-crafted templates, has potential to influence how future LLM-driven query improvement tools get built. For organizations relying on automated scanning to secure C/C++ codebases, a framework that boosts true positive detection while holding precision above 98% could mean fewer wasted engineering hours chasing false alarms and fewer real vulnerabilities slipping through unnoticed.
The discovery of new bugs in libpng and zlib also underscores something important: even mature, heavily scrutinized open-source libraries can still harbor undetected flaws, and better-refined detection queries can find them where existing tools couldn’t.
FAQ
What problem does ARQ aim to solve in C/C++ vulnerability detection?
ARQ addresses false positives and false negatives in existing CodeQL queries by automatically refining them using execution evidence from synthesized programs.
How does ARQ identify weaknesses in CodeQL queries?
ARQ detects query weaknesses whenever the execution of a synthesized program disagrees with the query’s verdict: a mismatch signals either a false positive or a false negative.
What role do large language models play in ARQ?
ARQ uses an LLM-based iterative refinement loop to repair queries based on execution feedback, without needing labeled data or vulnerability-specific templates.
What tangible improvements did ARQ show in refining CodeQL queries?
ARQ-refined CodeQL queries increased true positive detection by up to 119.8% with at least 98.0% precision, fixed three long-standing GitHub issues, and uncovered two new bugs in real-world libraries.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
