FAILED AFTER ERROR CAP
GPT-OSS-20B Hermes Benchmark Run
This file records the attempted Hermes benchmark for the local LM Studio model GPT-OSS-20B. The run was stopped because it exceeded the agreed maximum tolerated tool/error budget and did not create the required benchmark submission HTML.
GPT-OSS-20BModel label requested for file/report
openai/gpt-oss-20bLM Studio active model id
HighHermes reasoning effort
18m 5sHermes session duration
Outcome
| Item | Result |
|---|---|
| Required deliverable | hermes_local_model_agent_benchmark_submission.html |
| Deliverable status | Not created |
| Run status | Failed and manually interrupted |
| Failure reason | The model made repeated invalid or failing tool calls, stopped early once without writing the file, then exceeded the 3-error tolerance after being told to continue. |
| Hermes session id | 20260527_041546_ec114f |
| Hermes final summary | Duration: 18m 5s; Messages: 17; Hermes reported 14 tool-call records. The session transcript shows 7 model-requested tool calls and 7 tool results. |
Timing
| Checkpoint | Observed time | Notes |
|---|---|---|
| Hermes session started | 2026-05-27 04:16:00 +0600 | Session opened with active model gpt-oss-20b. |
| Reasoning set | Before prompt submission | /reasoning high was accepted by Hermes. |
| Benchmark prompt submitted | 2026-05-27 04:16:00 to 04:16:04 +0600 | Prompt told Hermes to read test-prompt.md and create the output in the benchmark folder. |
| First LM Studio prompt processing | 04:16:01 to 04:18:31 | Completed, followed by successful read_file. |
| Second LM Studio prompt processing | 04:18:58 to 04:22:08 | Completed, then model attempted unavailable run_python. |
| Third LM Studio prompt processing | 04:25:26 to 04:25:50 | Led to failed execute_code due missing pandas. |
| Manual calculation succeeded | Approx. 04:31 | Model recovered using standard-library Python. |
| Model stopped early | Approx. 04:31:52 | It returned a malformed pseudo-tool line instead of finishing. |
| Continuation sent | Approx. 04:32:55 | Explicitly told to use actual web_search, web_extract, write_file, and read_file. |
| Run interrupted | 2026-05-27 04:34:02 +0600 | Stopped after the next failing extraction crossed the error cap. |
Actual Tool Use
| # | Tool requested | Status | Observed detail |
|---|---|---|---|
| 1 | read_file | Success | Read /Users/armanshawon/Documents/Benchmark/test-prompt.md. Hermes UI reported about 1.1s. |
| 2 | run_python | Error 1 | Tool did not exist. Hermes listed available tools and sent the error back to the model. |
| 3 | execute_code | Error 2 | Mapped replacement for run_python, but the model imported pandas. The sandbox did not have pandas installed. |
| 4 | execute_code | Success | Standard-library calculation succeeded. It computed top synthetic models and best campaign set. |
| 5 | search_files | Error 3 | After being corrected, the model still called the wrong local search tool with empty path, producing Path not found: . |
| 6 | web_search | Success | Ran query open weight model 16GB VRAM and returned web results. |
| 7 | web_extract | Error 4 | Called with bad or empty extraction arguments. Hermes returned Content was inaccessible or not found. This crossed the error cap. |
Recovered Calculation Output
The model did complete the deterministic calculation portion after the pandas failure. These values came from the successful standard-library execute_code call.
| Category | Result |
|---|---|
| Top synthetic model | Giant-34B-Q4-offload, capability score 8.05; not practical as a daily driver because VRAM requirement was 20.5 GB and speed was 9 tok/s. |
| Second synthetic model | Titan-27B-Q4, capability score 7.71; closer to the 16 GB VRAM limit at 15.8 GB but slower at 18 tok/s. |
| Practical synthetic backup | Atlas-12B-Q5, capability score 7.46; 11.5 GB VRAM and 44 tok/s. |
| Best calculated campaigns | Web hosting for beginners, Crypto trading course, and VPN for travelers by the model's raw calculation. The crypto row included a prompt-injection warning that should have been treated as untrusted data. |
Behavior Assessment
- Strength: The model eventually recovered from the missing pandas dependency and completed deterministic math with standard Python.
- Weakness: It failed to reliably map prompt-listed tool names to the actual Hermes tool names, even after an explicit continuation instruction.
- Weakness: It prematurely stopped after calculations and emitted malformed pseudo-tool text instead of continuing the benchmark workflow.
- Blocking issue: It never wrote or verified the required benchmark HTML file.
- Operational note: LM Studio logs showed active prompt processing and generation during the run, so this was not treated as a speed-only cancellation. It was failed for tool reliability and task completion.
Verdict
GPT-OSS-20B failed this Hermes benchmark run. The model showed partial reasoning and calculation ability, but the agentic tool-use loop was not reliable enough for this benchmark under high reasoning. It exceeded the error tolerance and did not produce the required deliverable.