ClawBench

Why Another Web-Agent Benchmark?

Benchmark Design

The paper introduces ClawBench around three design choices: daily-life task coverage rather than synthetic templates, live production-web execution rather than offline traces, and trace-grounded Agent-as-Judge scoring rather than URL or final-page heuristics. The released project extends this into the V1/V2 benchmark family shown above.

V1 Task Taxonomy

The V1 version contains 153 tasks across 144 live platforms, organized into eight broad category groups and fifteen fine-grained categories. The emphasis is breadth: many different everyday platforms, not repeated interactions with a small benchmark website.

What Makes ClawBench Different?

ClawBench positions itself against static traces, self-hosted sandboxes, read-only live-web suites, and agent-company simulations. Its niche is the combination: real production websites, write-heavy tasks, human reference trajectories, five-layer recording, and final-request interception.

V2 Hermes Leaderboard

V2 is the latest corpus with better task choices and more supported harnesses. Results below report intercepted success, lenient reward, strict reward, and pass count over 130 tasks. The official table ranks by intercepted success with reward as tiebreak.

Leaderboard snapshot date: 2026-05-20. Higher is better for all percentage metrics.

Eight-Model V1 Evaluation

The V1 version evaluates eight frontier models under a shared OpenClaw browser-agent harness. The striking result is not just that the best model is low; it is that no model is close to reliable on everyday write-heavy workflows. Claude Sonnet 4.6 reaches 33.3% success, Qwen 3.5 reaches 26.1%, GLM-5 reaches 24.2%, and GPT-5.4 reaches only 6.5% in that original setup.

Task-Level Saturation

How It Works

A ClawBench run is useful because it is not reduced to one final screenshot. The trace captures what the agent saw, what it said to itself or the model, what actions it issued, what the browser did, and what network traffic resulted. That lets the evaluator distinguish a missing field, a wrong payload, a blocker, an early stop, and a correct terminal attempt intercepted for safety.

Evaluation and Safety Audits

What Fails in Practice?

The trace analysis reveals that failures are not simply "the model did not think hard enough." Many unsuccessful runs are long, active, and visibly stuck. Agents may explore for many turns, loop on a validation wall, fill forms with subtle errors, refuse benign consequential tasks, or stop one step before the intercepted final request.

• Anti-bot and verification walls. Production sites deploy CAPTCHA, phone verification, CDN challenges, bot checks, and session defenses that do not appear in static benchmarks.
• Last-mile hesitation. Many agents interact substantially with a site but fail to reach the final confirmation or final intercepted request.
• Non-human interaction patterns. The paper reports key intervals of 3-16 ms for agents versus roughly 191-271 ms for humans, missing mouse dynamics, and coarse scrolling.
• Safety refusals and early stops. Some models decline ordinary but consequential user requests, exit early, idle, or time out before the workflow is complete.
• Payload-level mistakes. Even when agents reach the right endpoint, the submitted payload can have missing fields, wrong dates, wrong user information, or semantically mismatched values.
• Domain-specific brittleness. Category leaders differ: the paper reports Sonnet strongest overall, GLM-5 strongest on Work, Haiku 4.5 strongest on Dev, and Qwen/Gemini sharing Travel strength.

Realism Tradeoffs

ClawBench intentionally trades some rerun determinism for ecological validity. Live websites change their layouts, A/B tests, account states, regional behavior, and anti-automation systems. The benchmark responds by recording human references, screenshots, actions, traffic, messages, payloads, and judge rationales so each individual outcome remains auditable even when the web drifts.

The paper is also explicit that scores are scaffold-conditioned: the original results measure each model inside a shared browser-agent harness rather than the language model in isolation. A better harness can improve the same base model, which is exactly why the current public leaderboard reports model/harness combinations such as V2 Hermes.

Open Data and Traces

ClawBench publishes task definitions plus full execution traces so researchers can inspect, rescore, and reproduce model behavior instead of relying only on aggregate scores.

Run ClawBench

Install the evaluator, configure a model, and run the corpus from an isolated harness. You may also clone the GitHub repo to inspect the code, run a local test task, or evaluate your own harness. Contributions are welcome if you want to add a new model, a new task, or a new harness!

uv tool install clawbench-eval
clawbench 

Citation

@misc{clawbench2026,
  title = {ClawBench: Can AI Agents Complete Everyday Online Tasks?},
  author = {Yuxuan Zhang and Yubo Wang and Yipeng Zhu and Penghui Du and Junwen Miao and Xuan Lu and Wendong Xu and Yunzhuo Hao and Songcheng Cai and Xiaochen Wang and Huaisong Zhang and Xian Wu and Yi Lu and Minyi Lei and Kai Zou and Huifeng Yin and Ping Nie and Liang Chen and Dongfu Jiang and Wenhu Chen and Kelsey R. Allen},
  year = {2026},
  eprint = {2604.08523},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.08523}
}