Hermes Local Model Benchmark Failure Report

FAILED: prompt template error before inference

This report records the attempted Hermes benchmark for Gemma-4-26B-A4B-IT-IQ2_M-Unsloth. The run did not reach model generation or benchmark task execution. LM Studio rejected the chat request while rendering the model prompt template, so Hermes could not read or execute /Users/armanshawon/Documents/Benchmark/test-prompt.md.

0. Benchmark Runner Metadata

Model labelGemma-4-26B-A4B-IT-IQ2_M-Unsloth

Loaded LM Studio identifierunsloth/gemma-4-26b-a4b-it

Hermes displayed modelgemma-4-26b-a4b-it

Reasoning efforthigh, confirmed by Hermes

Hermes session20260527_053654_47aa54

First benchmark prompt sent2026-05-27 05:38:15 +06

First failure confirmed2026-05-27 05:38:40 +06

Retry prompt sent after 96K reload2026-05-27 05:43:44 +06

Retry failure confirmed2026-05-27 05:44:09 +06

Final retry prompt sent after user fix2026-05-27 05:48:58 +06

Final retry failure confirmed2026-05-27 05:49:25 +06

Hermes session duration1m 50s

Provider/API elapsed timeAbout 8 seconds before final failure

Hermes message/tool count1 user message, 0 Hermes tool calls

LM Studio endpointhttp://127.0.0.1:1234/v1

LM Studio status after failureIDLE, model still loaded

1. What Was Attempted

Step	Observed result
Confirmed LM Studio model	`unsloth/gemma-4-26b-a4b-it` was loaded and idle with a 96,000-token context window.
Started Hermes	Hermes Agent v0.14.0 opened session `20260527_053654_47aa54` and displayed `gemma-4-26b-a4b-it`.
Set reasoning	`/reasoning high` succeeded and Hermes confirmed the reasoning effort was saved as `high`.
Sent benchmark instruction	Hermes was instructed to read `test-prompt.md`, run the benchmark, and create the output in the Benchmark folder.
Provider request	The request reached the local LM Studio endpoint but failed before token generation.

2. Failure Detail

Hermes made three provider attempts. Each failed with the same LM Studio template-rendering error:

API call failed after 3 retries: Error rendering prompt with jinja template:
"Cannot call something that is not a function: got UndefinedValue".

Hermes also wrote a request debug dump at: /Users/armanshawon/.hermes/sessions/request_dump_20260527_053654_47aa54_20260527_053828_323778.json

2a. Retry Detail

A second run was attempted in a new Hermes session. Before that retry, LM Studio was found to be advertising only 54,835 context tokens, which caused Hermes to refuse initialization because it requires at least 64,000 tokens. The same model was then reloaded through LM Studio with a 96,000-token context and the retry was started again.

Retry event	Result
New Hermes session before reload	`20260527_054234_1a2fd8` refused to initialize: advertised context was `54,835`, below Hermes' `64,000` minimum.
Same model reloaded	`unsloth/gemma-4-26b-a4b-it` was reloaded with `--context-length 96000`, `--parallel 4`, and the same identifier.
Clean retry Hermes session	`20260527_054329_cfb6bb`, reasoning `high`, 1 user message, 0 Hermes tool calls, 40s session duration.
Clean retry provider attempts	All 3 attempts again failed before inference with the same Jinja prompt-template error.
Clean retry debug dump	`/Users/armanshawon/.hermes/sessions/request_dump_20260527_054329_cfb6bb_20260527_054358_085439.json`

2b. Final Retry After Context Fix

After the user fixed the context configuration, a final clean run was attempted in a new Hermes session. LM Studio advertised a 90,000-token context, so Hermes initialized successfully. The provider still failed before inference with the same Jinja prompt-template error.

Final retry event	Result
LM Studio state	`unsloth/gemma-4-26b-a4b-it` loaded, idle, `90,000` context, `parallel 4`.
Hermes session	`20260527_054846_bb2516`, model `gemma-4-26b-a4b-it`, reasoning `high`, 38s duration.
Benchmark handoff	Prompt sent at 2026-05-27 05:48:58 +06 with instructions to run the same benchmark and write the result in the Benchmark folder.
Provider attempts	All 3 attempts failed before inference with `Error rendering prompt with jinja template: "Cannot call something that is not a function: got UndefinedValue"`.
Hermes tools	0 Hermes tool calls. The model never produced an actionable response because LM Studio rejected prompt rendering.
Final retry debug dump	`/Users/armanshawon/.hermes/sessions/request_dump_20260527_054846_bb2516_20260527_054912_636124.json`

3. Tool Call / Execution Detail

Tool or command path	Purpose	Result
LM Studio local server	Serve `unsloth/gemma-4-26b-a4b-it` through the OpenAI-compatible endpoint.	Server was available, but prompt rendering failed inside the model template before inference.
Hermes CLI	Run the benchmark agent using the active local model.	Started successfully, accepted reasoning setting, then failed on provider API retries.
Hermes agent tools	Expected to read files, browse, and create the benchmark artifact.	No Hermes tool calls occurred because the model never produced an actionable first response.
Benchmark supervisor checks	Confirm model state, endpoint availability, disk space, session status, and final idle state.	Model remained loaded and idle; disk had about 17 GiB free; no long-running generation was active.

4. Outcome

This run is marked failed. The final retry confirmed the failure persists even after the context issue was fixed and Hermes initialized with a 90K context. The failure was not caused by generation speed, response quality, context length, or benchmark task performance. It was caused by a local model prompt-template incompatibility in LM Studio for the loaded unsloth/gemma-4-26b-a4b-it variant.

Practical implication: this model variant needs a working LM Studio chat template or a corrected/lmstudio-community build before it can be benchmarked through Hermes.