Screenshot and DOM Observation Pipelines for Computer-Use Agents

Reliable agents depend less on what they see than on how well they ground vision into action.

· 10 min read
Cover illustration for “Screenshot and DOM Observation Pipelines for Computer-Use Agents”
Browser Agent Architecture · October 9, 2026 · 10 min read · 2,258 words

A computer-use agent's reliability lives almost entirely in its observation pipeline. Every one of these systems runs the same four-step loop, observe, ground, act, verify, and the first two steps determine almost all of the system's reliability.

The observe step captures a representation of the current state of the screen: a screenshot, a DOM snapshot, an accessibility tree, or some combination of the three. The ground step takes that representation and maps a natural-language intent, "click the submit button," "select row 14," onto a concrete, actionable target: a pixel coordinate, an element reference, an accessibility node. The act step then issues the actual input event, a click, a keystroke, a scroll, a drag, through the operating system's input layer or a browser automation protocol such as Chrome DevTools Protocol or Playwright. Verify closes the loop: the agent observes again, confirms whether the prior action produced the intended effect, and decides whether to continue, retry, or stop.

Grounding is where almost everything goes wrong. The gap between clicking the right cell in a spreadsheet and clicking the cell just to its left looks trivial on paper and is often the entire difference between a working agent and a broken one. Every observation method examined in this piece earns its place, or fails to, based on how well it feeds the grounding step, and on how cheaply it lets the agent verify its own actions before it commits to the next one.

The two architectural families and what each can see

Before any individual observation method matters, an agent's builders make a prior architectural commitment: does the agent operate at the level of the whole desktop, or only inside a browser context? That decision determines which observation signals are even available.

OS-level agents control the entire desktop. They can reach any application, native file dialogs, terminal windows, legacy desktop clients, anything rendered on screen. They observe through screenshots combined with the operating system's own accessibility APIs: UI Automation (UIA) on Windows, the Accessibility API (AX) on macOS, and AT-SPI on Linux. This reach comes at a cost. It tends to fail when the model clicks the wrong target because it has not seen structured data for a widget, like a custom slider, a canvas-rendered chart, or an unlabeled icon. OS-level agents suit back-office work that spans multiple applications at once: desktop ERP and CRM systems, document workflows, anything that bounces between a native client and the file system.

Browser-level agents operate inside a single browser context. They observe the DOM and the accessibility tree through the Chrome DevTools Protocol or through Playwright's own accessibility APIs, both giving structured, symbolic information about the page. This narrower scope buys speed, lower cost, and more deterministic behavior: selectors tend to be stable, and sessions can run in parallel across many lightweight cloud instances. The tradeoff is blindness outside the tab. If you need web research, form filling at scale, scraping, or SaaS-only workflows that never leave the page, browser-level agents are the right tool.

This fork is an infrastructure decision made before a single agent runs, not a matter of picking a different model. OS-level agents require a persistent, full-desktop virtual machine to operate against, with all the standing compute cost that implies. Browser-level agents can run inside lightweight, isolated browser contexts that spin up and tear down per task. That difference carries direct consequences for deploying agents per user: keeping a full persistent desktop environment alive at rest for every individual user gets expensive fast, which is part of why snapshot-restore scheduling for those environments becomes a serious design concern in its own right, a topic that deserves its own full treatment.

How screenshots, DOM, and accessibility trees contribute to grounding

No single observation method covers the whole problem. Each one solves a different piece of the grounding puzzle and carries its own cost structure, and the choice between them is an economic decision as much as a technical one.

Screenshots are the universal fallback. Every interface, no matter how it's built, produces pixels: legacy desktop software with no accessibility metadata, canvas-rendered dashboards, custom widgets with no corresponding DOM node or accessibility entry. A screenshot sees all of it. A raw screenshot sent at every step tends to be the single largest cost driver in a multi-step agent run, which compounds the token cost. A single screenshot also only captures a moment: it misses the animation, the loading spinner, or the transition happening between one action and the next.

DOM access gives the agent something closer to a direct address book: element IDs, classes, attributes, and text content it can reference symbolically, without inferring a coordinate from a picture. When a page's DOM is clean and complete, this is cheap and precise. It breaks down when elements are visually present but structurally absent, rendered dynamically, or tucked behind canvas or iframe boundaries the DOM doesn't describe. DOM access is also blind to layout in a specific way: two elements that sit next to each other in the DOM tree can be rendered far apart on the actual page, and no amount of DOM inspection tells the model that without pixel evidence to confirm it.

Accessibility trees sit a step closer to intent. They expose the semantic role hierarchy that the operating system or browser already maintains for screen readers and other assistive technology, so they stay more stable than raw coordinates, and they often tell you more than a bare DOM node does. On the desktop, the OS accessibility APIs, UIA, AX, AT-SPI, are the only structured alternative to screenshots for native applications that have no DOM. Their weakness is that they depend entirely on developers having maintained accurate ARIA roles and labels, and in a lot of enterprise SaaS software that work is incomplete or missing, so the tree ends up incomplete or misleading exactly where an agent needs it most.

Set-of-Marks overlays try to bridge pixel and symbolic grounding directly: interactive elements get numbered or labeled right on top of the screenshot, giving the model a visual index to point at instead of inferring a raw coordinate. This works well on sparse pages with a handful of clear targets. But if a page is dense, with hundreds of tightly packed elements, the marks themselves clutter and occlude the view they're meant to clarify. Probe to Act (P2A) identifies the deeper limitation here: Set-of-Marks forces a static, global alignment, every mark placed at once, before the model has decided anything, with no mechanism built in to verify an ambiguous candidate or recover a target that never had a DOM entry to begin with.

All of these methods are ultimately bounded by the training signal behind the model interpreting them. The GroundNext model series, trained at 3B and 7B parameter scales on the GroundCUA dataset using supervised fine-tuning and reinforcement learning, works toward zero-shot instruction-following across desktop, web, and mobile interfaces by combining vision, language, and action signals rather than leaning on any one observation channel alone.

What static single-method observation forces on real-world pages

The failure here is the static, one-shot way agents apply these methods: observe once, from one modality, commit to a target, act. That rigidity is what breaks on real pages.

Dense interfaces expose this cleanly. A page with hundreds of tightly packed elements overwhelms screenshot-based coordinate inference and overwhelms a static overlay that annotates candidate elements directly on the image at the same time, for the same reason: both approaches ask the model to resolve every candidate target in a single static pass, with no way to narrow down ambiguity before committing to an action. DOM and pixel evidence disagree with each other in an asymmetric way that makes this worse. Neither source resolves the conflict on its own, and a static pipeline that reads one or the other, or both, without a mechanism to adjudicate between them has no way to know which signal to trust in the moment it matters.

If an agent resolves this alignment statically, before the decision instead of during it, it can't check an ambiguous candidate before it acts on it. A mis-click under that architecture doesn't just waste a step. It commits to an irreversible state change, a form submitted, a record deleted, a wrong recipient selected, that the verify step then has to detect after the fact rather than the ground step preventing before it happens.

Long-horizon tasks make the economics of this worse, not just the accuracy. Research on Probe to Act finds that keeping full observation history across a long task runs to roughly three times the context size of keeping only the action history, and that multiplier compounds cost and latency the longer a task runs.

A temporal gap sits beneath these failures, produced by the fact that a single static frame can't show the process that occurred between actions. None of this is yet standard in production pipelines, but it names a real structural gap: an agent that only ever looks at one frame per action is structurally unable to see the process that frame was the output of. Fixing any of this doesn't come from a richer static snapshot, a bigger screenshot, a denser annotated overlay, a longer DOM dump. It requires the agent to align modalities on demand, at decision time, not before it.

How active, hybrid observation pipelines resolve the grounding bottleneck

The shift from static to active observation, where the agent itself decides what to align, when to align it, and which modality to trust for a given decision, is the most consequential architectural move happening in production observation pipelines right now.

Probe to Act (P2A), introduced by Li et al. from the Institute of Computing Technology at the Chinese Academy of Sciences and StepFun, is the clearest working example. Its core mechanism runs before any state-changing browser operation gets committed: the agent issues lightweight probes that translate a DOM handle into pixel evidence, map a screen region back to its DOM candidates, register a visual-only target that has no DOM backing at all, and commit a verified note to memory once a target is confirmed. This takes DOM-pixel alignment out of static preprocessing and turns it into an active reasoning step interleaved directly with the agent's decision-making, rather than a pass completed once before the model ever reasons about the task. P2A's memory is evidence-based: only observations that were actually probed, acted on, or explicitly committed get retained across steps, which is how the system matches the context efficiency of full-observation history while retaining far less input overall. On VisualWebArena, P2A improves task success for both proprietary and fine-tuned models, lifting Gemini-3-Pro from 54.1% to 61.2% and Qwen3-VL-8B from 24.6% to 32.9%.

Token-efficient, screenshot-first pipelines following the Aguvis pattern take a different route to the same goal: when the interface is visually unambiguous and the DOM is noisy, incomplete, or actively misleading, the right call is to drop the redundant text surrogate altogether and ground directly from the image. This isn't a weaker fallback. When DOM evidence would add cost without adding accuracy, this is the correct choice.

You can distinguish a separate pattern from both of the above: the compiled state machine, as seen in SkillDroid, built for repetitive mobile interactions where per-step visual inference from a large model introduces serving latency that a production system can't absorb on every repeated action. This is an optimization layered on top of an observation pipeline, not a substitute for one. The underlying observation and grounding pipeline still runs in full on any path the agent hasn't seen before, so compiled transitions only bypass that pipeline on paths already known to work. It's a caching layer, and a valuable one, sitting above the grounding problem.

The principle shared across P2A, screenshot-first pipelines, and compiled caching layers is the same: push the alignment decision from preprocessing time to decision time, and retain only the evidence the agent actually consulted along the way. That combination is what cuts token cost and mis-grounding at once, something a static pipeline structurally cannot do, since a richer static overlay only adds more cost and more visual clutter without resolving the underlying alignment problem it was meant to fix.

Diagram: Active Probing Lifts Task Success Across Models. Visualizes: Show the before-and-after task success rates for two models when Probe to Act (P2A) is added, as measured on VisualWebArena.

What benchmarks measure and where observation pipelines fall short

Benchmarks for computer-use agents each measure a different slice of the observation-grounding problem, and no single one of them captures the full range of failure modes a production system will eventually run into.

GroundCUA, paired with the GroundNext model series, presented at ICLR 2026, is built specifically to isolate the grounding step: given a screenshot and a natural-language instruction, can the model correctly identify the target element the instruction refers to? What it measures well is grounding accuracy in isolation. What it doesn't measure is the full loop: a model can ground correctly against a static screenshot and still fail in production if the page has changed state by the time the act step fires, or if the task requires reasoning across a long sequence of steps rather than a single instruction-to-target mapping.

That gap between benchmark and production is the honest place to end on. Active hybrid pipelines like P2A show measurable gains on benchmarks such as VisualWebArena, and grounding-specific datasets like GroundCUA show real progress on isolating and improving the grounding step itself. Neither constitutes a solved problem. Long-horizon planning, precise localization under ambiguity, and state-dependent interaction, where the correct target depends on what the page did a few steps earlier, remain the open engineering challenges that no current benchmark fully captures and no current pipeline fully resolves.

Sources

  1. Probe to Act: Elevating Browser-Use Agent via Active Visual Probing
  2. GROUNDING COMPUTER USE AGENTS ON HUMAN ...

More in Browser Agent Architecture