Ministral-3-14B-Reasoning-MistralAI Hermes Benchmark Failure Report
Result: Failed / partial run
The model completed the initial prompt read and wrote a markdown planning report, but it did not produce the required benchmark HTML file. After a long generation period with no new file output, I interrupted the stalled finalization turn and asked Hermes to continue. The continuation then failed three times on the same LM Studio prompt-template error.
Outcome
| Requirement | Status | Detail |
|---|---|---|
| Confirm active model | Passed | Hermes showed ministral-3-14b-reasoning; LM Studio showed mistralai/ministral-3-14b-reasoning. |
| Set reasoning to high | Passed | Hermes accepted /reasoning high and confirmed the setting. |
| Read benchmark prompt | Passed | Hermes read /Users/armanshawon/Documents/Benchmark/test-prompt.md. |
| Create required HTML artifact | Failed | No fresh hermes_local_model_agent_benchmark_submission.html or model-specific final HTML was created by Hermes for this run. |
| Recover after stall | Failed | A continuation prompt triggered repeated LM Studio template failures. |
Timeline
| Time | Event |
|---|---|
| 2026-05-27 05:56:40 +0600 | Fresh Hermes session opened. Session key: 20260527_055640_dfa588. |
| 2026-05-27 05:56:55 +0600 | Benchmark instruction sent for mistralai/ministral-3-14b-reasoning. |
| Early run | LM Studio moved through prompt processing and active generation. Hermes context reached about 21.3K / 78K after prompt read. |
| About 3 minutes elapsed | Hermes read the benchmark prompt file successfully. |
| 2026-05-27 06:05 +0600 | Hermes wrote Ministral-3-14B-Reasoning-MistralAI-benchmark-report.md, but it was only a planning report, not the required benchmark output. |
| About 14m 46s elapsed | Generation had continued for 368.7 seconds after the last file write without producing the final HTML. I interrupted the API call. |
| 2026-05-27 06:12:05 +0600 | Continuation attempt failed after three built-in Hermes retries with the same Jinja prompt-template error. |
| 2026-05-27 06:12:31 +0600 | Failure report prepared after closing the Hermes session. |
Tool Call Audit
| Order | Hermes tool | Target | Observed result |
|---|---|---|---|
| 1 | read_file |
/Users/armanshawon/Documents/Benchmark/test-prompt.md |
Completed. TUI showed about 1.1s. |
| 2 | write_file |
/Users/armanshawon/Documents/Benchmark/Ministral-3-14B-Reasoning-MistralAI-benchmark-report.md |
Completed. 1,943 bytes written. TUI showed about 0.8s. |
| 3 | Continuation model call | Attempt to produce Ministral-3-14B-Reasoning-MistralAI-high.html |
Failed before any tool call because LM Studio rejected the prompt template. |
Error Details
The original long generation did not show a provider error; LM Studio reported GENERATING while no new file appeared.
After interruption and continuation, Hermes recorded three API failures from the local provider.
Provider: custom Model: mistralai/ministral-3-14b-reasoning Endpoint: http://127.0.0.1:1234/v1 Error: Error rendering prompt with jinja template: "After the optional system message, conversation roles must alternate user and assistant roles except for tool calls and results." Retries: 3/3 exhausted Request debug dump: /Users/armanshawon/.hermes/sessions/request_dump_20260527_055640_dfa588_20260527_061205_530413.json
Partial Artifact
Hermes created this partial markdown report:
/Users/armanshawon/Documents/Benchmark/Ministral-3-14B-Reasoning-MistralAI-benchmark-report.md.
The markdown explicitly says the HTML file still needed to be generated, so it is not a valid benchmark completion artifact.
Judgment
This model run should be recorded as a failure for the benchmark. It demonstrated basic tool use by reading the prompt and writing a small markdown file, but it failed the required deliverable and could not recover after interruption because of a model prompt-template incompatibility in LM Studio.