Qwen3-Coder-30B-A3B-Instruct-Q3_K_S Hermes Benchmark Failure Report

Result: Failed before generation due to LM Studio prompt-template error

This run did not reach the benchmark task. Hermes confirmed the loaded model and set reasoning to high, but every API attempt failed while LM Studio rendered the model prompt template. No file was read, no model output was generated, and no benchmark artifact was created by Hermes.

Model label

Qwen3-Coder-30B-A3B-Instruct-Q3_K_S

LM Studio identifier

qwen3-coder-30b-a3b-instruct

Hermes displayed model

qwen3-coder-30b-a3b-instruct

Quantization noted by user

Q3_K_S

Reasoning effort

high

LM Studio context

200,000 tokens

LM Studio size / parallel / device

13.29 GB / 4 / MGPC

Hermes session key

20260527_063705_6a1a88

Hermes session duration

1m 13s

API endpoint

http://127.0.0.1:1234/v1

Run Summary

Checkpoint	Status	Evidence
Confirm loaded model	Passed	LM Studio showed `qwen3-coder-30b-a3b-instruct` loaded and idle with a `200000` token context. Hermes displayed the same model name in the session header.
Set reasoning high	Passed	Hermes confirmed `Reasoning effort set to 'high' (saved to config)`.
Submit benchmark prompt	Passed	The prompt instructed Hermes to use `Qwen3-Coder-30B-A3B-Instruct-Q3_K_S`, read `/Users/armanshawon/Documents/Benchmark/test-prompt.md`, and create the required result file in the Benchmark folder.
Model API call	Failed	All 3 API attempts failed before generation with `Unknown StringValue filter: safe` while rendering the Jinja prompt template.
Read `test-prompt.md`	Not reached	Hermes exited with `0 tool calls`; no file-read tool was called.
Create benchmark HTML	Not reached	No benchmark artifact was written by the tested model. This failure report was created by the supervising Codex run after termination.

Error Details

APIError
Provider: custom
Model: qwen3-coder-30b-a3b-instruct
Endpoint: http://127.0.0.1:1234/v1
Error: Error rendering prompt with jinja template: "Unknown StringValue filter: safe".

Hermes reported that this is usually caused by the model's prompt template. It suggested using an LM Studio community build with a fixed prompt template or otherwise correcting the template before retrying.

Attempts

Attempt	Elapsed shown by Hermes	Context shown by Hermes	Result
1 of 3	0.30s	2 messages, about 8,700 tokens	Prompt-template render error
2 of 3	2.67s	2 messages, about 8,700 tokens	Prompt-template render error
3 of 3	7.44s	2 messages, about 8,700 tokens	Prompt-template render error; retries exhausted

Timeline

Time	Event
2026-05-27 06:37:05 +0600	Fresh Hermes session opened. Session key: `20260527_063705_6a1a88`.
Start of run	LM Studio confirmed `qwen3-coder-30b-a3b-instruct`, size `13.29 GB`, context `200000`, parallel `4`, device `MGPC`.
Before benchmark prompt	Hermes confirmed `/reasoning high`.
During API attempts	Hermes made three API attempts through the custom OpenAI-compatible endpoint. All failed on the same Jinja template error before generation began.
2026-05-27 06:37:30 +0600	Hermes wrote a debug dump at `/Users/armanshawon/.hermes/sessions/request_dump_20260527_063705_6a1a88_20260527_063730_167618.json`.
End of run	Hermes session exited after `1m 13s` with `1` user message and `0` tool calls.

Tool Call Detail

Tool or command	Purpose	Result
`hermes`	Started a fresh Hermes CLI session.	Succeeded; model header showed `qwen3-coder-30b-a3b-instruct`.
`/reasoning high`	Set Hermes reasoning effort to high before the test prompt.	Succeeded; Hermes saved the setting.
Benchmark prompt	Requested the model-specific test and asked Hermes to read `test-prompt.md`.	Failed before the model could respond because LM Studio rejected the rendered prompt template.
Hermes model tool calls	Expected file read and result-file creation actions.	None occurred. Hermes summary showed `0 tool calls`.

Judgment

Fail this run as an environment/template compatibility failure. The model was loaded and reachable, but LM Studio could not render its active chat template because the template used an unsupported safe filter. This should be fixed in LM Studio by changing or repairing the model prompt template before retesting.