A new study reveals that the very way artificial intelligence models click on your screen could be turned against them. Researchers have found a previously overlooked weakness in how GUI visual grounding models — the AI systems that translate on-screen instructions into actual clicks — generate coordinates, and they’ve built a working exploit called MissClick to prove it. The findings, published by researcher Yu Ran, expose a new category of adversarial attacks GUI models face as they become more embedded in automating tasks on desktop, web, and mobile platforms.
Summary
Key takeaways
- GUI visual grounding models generate screen click coordinates as sequences of digit tokens, not as raw numbers.
- A single-digit change in a high place value, like the hundreds place, can shift a click by 100 units on screen.
- MissClick is a white-box attack with two variants: MissClick-U for untargeted disruption and MissClick-T for targeted hijacking.
- MissClick-U reached untargeted success rates of 75.07% on OS-Atlas and 72.93% on UGround.
- MissClick-T reached targeted success rates of 44.86% on OS-Atlas and 62.67% on UGround.
How GUI Visual Grounding Models Generate Coordinates
GUI visual grounding models predict where to click by producing digit-by-digit outputs rather than direct numerical coordinates, and that seemingly small design choice turns out to carry real security consequences. According to the research, these models were never built with adversarial resistance to digit-level manipulation in mind, which is exactly the gap MissClick exploits.
Digit-Serialized Coordinate Prediction
Instead of outputting an X-Y coordinate as a single number, these models generate it as a sequence of individual digit tokens that get parsed afterward into a usable numerical value mapped to an actual point on the screen. Each digit token is essentially treated by the model as a categorical choice, similar to picking a word from a vocabulary, rather than as part of a continuous number.
Impact of Digit Place Value on Coordinate Displacement
That distinction matters enormously once the digits are parsed back into numbers. The study found that changing a hundreds-place digit by just one value can shift the corresponding coordinate component by 100 units once it’s converted into a screen position. In practical terms, a tiny, almost invisible tweak to a single token can send a click drifting far from its intended target — an inconsistency between how the model “thinks” about digits and how those digits ultimately behave once turned into real pixels.
MissClick: Adversarial Attacks on GUI Grounding Models
Building directly on that discovery, the researcher introduces MissClick, a white-box adversarial attack designed to exploit the mismatch between categorical digit prediction and numerical coordinate output. Unlike generic adversarial attacks GUI models have faced before, MissClick is purpose-built around the place-value structure of digit-serialized coordinates rather than treating the output as ordinary text.
Attack Variants: MissClick-U and MissClick-T
MissClick comes in two distinct flavors depending on the attacker’s goal. MissClick-U is the untargeted version, aimed simply at making the model click somewhere wrong — anywhere outside the correct region counts as success. MissClick-T is the targeted version, engineered to steer the click into a specific, attacker-chosen area instead of just derailing it randomly.
Distinct Attack Objectives and Optimization Strategies
The two variants rely on different mathematical objectives because untargeted and targeted disruption require different success conditions. MissClick-U works by maximizing what the study calls soft-coordinate displacement, essentially pushing the predicted click as far as possible from its correct location. MissClick-T instead minimizes a place-weighted target-digit loss, a technique that specifically prioritizes manipulating higher place-value digits since those produce the largest jumps once the coordinate is parsed. This targeted-versus-untargeted split reflects a broader point in GUI visual grounding security research: attackers optimize differently depending on whether they want chaos or precision.
Experimental Evaluation and Attack Success Rates
The two MissClick variants were tested against existing attacks on two established benchmarks, OS-Atlas and UGround, which cover desktop, web, and mobile grounding tasks. The results show that this digit-serialized coordinate vulnerability is not a marginal edge case — it’s exploitable at scale, with both attack types substantially outperforming prior methods.
Untargeted Attack Performance on OS-Atlas and UGround
MissClick-U posted untargeted success rates of 75.07% on OS-Atlas and 72.93% on UGround, improvements of 16.62 and 30.72 percentage points over existing attack baselines, respectively. Those are the highest scores among the objectives tested for untargeted disruption, and the study attributes that edge specifically to the soft-coordinate displacement approach.
Targeted Attack Performance on OS-Atlas and UGround
MissClick-T’s numbers tell a similarly striking story on the targeted side, with success rates of 44.86% on OS-Atlas and 62.67% on UGround, gains of 31.73 and 47.06 percentage points over prior methods. The MissClick attack success rates reported here confirm that place-weighted target-digit optimization is the strongest available strategy when an attacker wants precision rather than pure disruption, since it consistently beat other targeted objectives tested in the study.
Taken together, the gap between untargeted and targeted performance highlights a practical tension: hijacking a click into a specific spot is inherently harder than just making a model miss, because it demands controlling multiple digit places simultaneously rather than simply nudging one number outward. That asymmetry could shape how future defenses get prioritized, since a model resistant to random misfires isn’t necessarily resistant to precise redirection — a distinction that matters a great deal for any system relying on automated clicking to complete real tasks.
FAQ
What is the main vulnerability exploited by MissClick in GUI grounding models?
MissClick exploits the digit-serialized coordinate generation process where each digit is a categorical token, and changes in high place-value digits cause large coordinate displacements once those tokens are parsed into numbers.
What are the two variants of the MissClick attack and their goals?
MissClick-U is an untargeted attack that maximizes coordinate displacement to disrupt clicks generally, while MissClick-T is a targeted attack that minimizes a place-weighted target-digit loss to hijack coordinates into attacker-specified regions.
How effective are MissClick attacks on different datasets?
MissClick-U achieved untargeted success rates around 75% on OS-Atlas and 73% on UGround, whereas MissClick-T achieved targeted success rates of about 45% on OS-Atlas and 63% on UGround.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
