Evaluate Hermes Browser Tasks with ClawBench
Evaluate whether Hermes can complete real browser tasks: ClawBench runs each task in an isolated container with an instrumented Chromium, intercepts critical requests before they reach live services, and records traces for scoring and diagnosing failures.
Requirements
- Prerequisites are Python 3.11+,
uv, and Docker or Podman. - You add a model endpoint and API key to
models/models.yaml, then use that entry's name in the run commands. - Python 3.11+ and
uv. - Docker or Podman.
- An API endpoint supported by ClawBench's model configuration.
How to use
No separate Hermes installation is needed: selecting --harness hermes installs the pinned Hermes version in the harness image, attaches its native browser tools to the Chromium instance over CDP, and passes the chosen task instruction to hermes chat.
The first run builds the Chromium and Hermes container image; later runs reuse it.
Results land under test-output/<model>/ with the five trace layers plus interception and run metadata.
--no-judge reports interception-only results; configure the documented judge model for payload-aware Stage 2 scoring.
The tasks use live websites, so layouts, availability, bot defenses, and alternate flows can change between runs.
Complete the following setup.
git clone https://github.com/TIGER-AI-Lab/ClawBench.git
cd ClawBench
cp models/models.example.yaml models/models.yaml
Add a model endpoint and API key to models/models.yaml, then use that entry's name in the commands below. Run one task without the optional LLM judge:
Run one task without the optional LLM judge:
uv run clawbench-run \
test-cases/v1-lite/142-office-secretary-tasks-collab-trello \
<model-name> \
--harness hermes \
--no-judge
Run the 20-task Lite suite:
uv run clawbench-batch \
--models <model-name> \
--cases-suite v1-lite \
--all-cases \
--max-concurrent 1 \
--harness hermes \
--no-judge
Remove --no-judge after configuring the judge entry documented in models/models.example.yaml.
Prompts: no separate Hermes prompt is required.
ClawBench uses the selected task's public instruction field as the base Hermes query. For the single-task command above, that instruction is:
Create a Trello board "Q3 Sprint Planning" with 3 lists + 5 cards
Source: aliaihub/awesome-hermes-usecases and its contributors. License: CC-BY-4.0. Source license notice.
Translation and editorial adaptation by ClawCases.
>Resources
>Related Use Cases
Agent Merge-Conflict Arbiter
Resolve a git merge conflict between two agents' branches as an impartial third party: a neutral arbiter that takes both diffs and their stated intents and produces a merged result without overwriting either side.
Autonomous Project Management with Subagents
Coordinate projects autonomously: a main agent delegates work to PM subagents that track tasks in a shared STATE.yaml, update it as they progress, pick up unblocked work in parallel, and keep all task state in version-controlled files.
Voice Assistant Workflows
Set up voice input and output for the Hermes agent so you can talk to it through a terminal microphone loop, Telegram or Discord voice notes, or a Discord voice channel and get spoken replies — all with the same agent, memory, and skills.