KNOWS

Tasks

A KNOWS’ task asks an agent to research something on the live web and turn what it finds into a finished Google Doc, Sheet, or Slide deck — satisfying content, formatting, and layout requirements at once. Every task is made of three parts.

• A task prompt. A detailed description in English of what the agent has to do, written to be clear enough that a detail-oriented human could complete the task with no further instruction.
• A set of checkpoints. Discrete, measurable milestones, each containing multiple separate evaluation steps. Tasks score on overall or partial completion.
• An evaluator. A script that automatically checks the artifact at each checkpoint, combining hard-coded logic, utility functions such as the Google Workspace APIs, and LLM and VLM judgments.

Task creation overview

Task desiderata

We translate our core design requirements into a rubric that guides task creation and keeps new tasks aligned with our motivation. Every task must satisfy all five.

• Realism. The task is something a real user might want to do, and would save time if automated.
• Information retrieval. Completing it requires browsing the live web for at least five pieces of information — often many more — drawn from at least two websites.
• Artifact synthesis. The result is a cohesive, well-organized Google Doc, Sheet, or Slide deck. Retrieved information must be meaningfully synthesized, not copied in unedited.
• Visual or spatial reasoning. Identifying or editing specific elements, honoring spatial relationships (left, right, above), or color-based modification — for example, remove all red links.
• Complexity. At least four steps of information retrieval, multiple steps to construct the artifact, and at least ten steps to incorporate and synthesize what was found.

Task statistics

Hybrid Evaluation Framework

For each evaluation step in a task, there is a verifier for the successful completion of that step. Each task has a dedicated evaluator script, which programmatically checks completion of each evaluation step, and returns completion scores for each checkpoint. Our evaluators are implemented by examining Google Workspace tool states, analyzing agent trajectories, and extracting data from the live web.

Eval step taxonomy

Many evaluation steps share common requirements in implementation, so we provide a suite of shared evaluation utilities available to any evaluator, organized around seven categories of evaluation step.

• Structural checks verify the existence, order, and hierarchy of elements via the Google Workspace API.
• Information retrieval checks compare extracted data values against gold-standard references using exact matching, fuzzy matching, or numerical comparison with tolerances.
• Formatting checks verify visual styling properties such as font weight, size, color, and alignment by querying the API directly.
• Spatial checks validate the physical layout of elements through their position, size, alignment, and whether they overlap or extend beyond the page.
• Content checks assess source fidelity and semantic accuracy of generated text.
• Web visit checks confirm that the agent actually visited relevant websites by inspecting its trace of visited URLs.
• Visual checks verify image content and provenance — whether an inserted image depicts the expected subject, originates from the correct source, or is visually distinct from other images.

Interactive

Inspect the evaluators yourself

Below you can see how our evaluators grade expert-created “gold standard” artifacts, compared to how artifacts created for the same task by the top-performing computer-use agent, Comet1, are graded.

Show the task prompt

Gold standard expert

Comet agent

Only steps where they differ ✓ passed 6/10 partial credit ✗ failed – not evaluated

Evaluation stepGoldComet

1 Comet running a Claude Opus 4.7 agent.

Benchmark Results

We evaluate state-of-the-art computer-use agents across open-source and proprietary vision-and-language models and harnesses. Scores are reported per artifact type and overall, on four metrics ordered from strictest to loosest.

Metrics

• SR — Success Rate. A task scores 1 only if every evaluation step across all checkpoints is correct. The standard complete-success metric.
• ASC — Average Successful Checkpoints. The fraction of a task's checkpoints in which all steps are correct. Each checkpoint is a milestone in building the artifact, so ASC counts milestones fully reached.
• ACF — Average Checkpoint Fraction. The fraction of steps correct within each checkpoint, averaged across checkpoints. Awards partial credit per milestone.
• SF — Step Fraction. The fraction of all evaluation steps correct. Similar in strictness to ACF, but weights complex checkpoints more heavily by pooling steps rather than averaging over checkpoints.

Systems

• BrowserGym–AgentLab, text only (AXT). The open-source harness driving each model from the accessibility tree alone — no screenshots. Run with GPT 5.5, Claude Opus 4.7, and the open-source DeepSeek V4 Pro.
• BrowserGym–AgentLab, multimodal (AXT + SS). The same harness given screenshots alongside the accessibility tree. Visual inputs were not available for the DeepSeek V4 Pro API at the time of writing, so it appears in the text-only block only.
• AI browsers. Closed-source harnesses that browse on their own: ChatGPT Atlas (GPT agent, selected automatically by the harness) and Perplexity Comet (Claude Opus 4.7 agent).

Results and analysis

• Long-horizon tasks remain challenging. SR is near zero across most methods and splits, showing both the difficulty of our benchmark and the inability of SOTA agents on challenging, long-horizon tasks. The exception is Docs, where the best baseline, Comet, achieves a non-zero, albeit low, SR (12.0%). This result is unsurprising, as doc tasks resemble the conversational, text-only outputs best suited for the underlying LLMs. Partial success is also similarly low across most methods. Among the general agents, GPT-5.5 performs relatively well compared to other models. While Comet achieves the best performance, qualitative results show many of these outputs are often unusable.
• Visual modality is important. KNOWS contains a large fraction of tasks that require visual and spatial reasoning, which are often unsolvable from the AXT alone. Adding visual inputs as screenshots (SS) with the AXT can significantly improve performance, as much as (+10.3 ASC, +25.5 ACF, +25.5 SF) for Opus 4.7 over its AXT-only variant.
• Harness matters as much as the model. Perhaps the starkest finding is the gap between two systems sharing the same backbone, e.g., Opus 4.7 with the Comet harness outperforms the BrowserGym harness by (+2.7 SR, +18.7 ASC, +27.5 ACF, and +25.6 SF) overall. Proprietary harnesses show clear advantages in live web browsing, e.g., on complex webpages and with pop-up advertisements.
• Qualitative error analysis. While the best-performing agents can achieve decent partial success scores, several of the resulting outputs often remain practically unusable. We observe similar errors in structuring documents, applying formulae in spreadsheets, structuring tables, and creating proper visualizations.

Acknowledgments

We thank Google for providing funding for the construction of this benchmark. We thank Anthropic and OpenAI for their donation of API credits for benchmarking models. We thank Ivan Andhika, Matthew Lee, Ryhor Pryslopski, Spike Cheng and Qui Ngo for their contributions to the codebase, Shrusti Ghela and Ana Alvarez Lopez for providing valuable annotations, and everyone who consented to our use of their information in this benchmark. We also thank everyone involved in the review process for providing valuable feedback on KNOWS.

BibTeX

Placeholder — the final entry will be updated once the Findings of EMNLP 2026 proceedings are published.

@inproceedings{gill2026knows,
  title     = {The Hard Part Comes After Search: Benchmarking Web Agents on
               Synthesizing, Organizing, and Displaying Knowledge},
  author    = {Gill, Alexander and Ishmam, Md Farhan and Nguyen, Xuyen and
               Bhat, Neha and DeYoung, Parker Henry and
               Hashemi Chaleshtori, Fateme and Stringham, Nathan and
               Marino, Kenneth and Marasovi\'{c}, Ana},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}