Qwen3.6-27B-MTP-IQ2_M Hermes Benchmark Failure Report

Result: Failed by user request due to unusable speed

This run was stopped because it was the slowest model observed so far in the local Hermes benchmark. The model spent several minutes processing the prompt, then continued generating without producing any usable Hermes-visible response, tool call, or benchmark artifact. The model was then ejected while the request was active, causing LM Studio to cancel the operation.

Model label

Qwen3.6-27B-MTP-IQ2_M

LM Studio identifier

qwen3.6-27b-mtp

Hermes displayed model

qwen3.6-27b-mtp

Quantization noted by user

IQ2_M

Reasoning effort

high

LM Studio context

69,000 tokens

LM Studio size / parallel / device

12.87 GB / 4 / MGPC

Hermes session key

20260527_064621_bfa0e3

Hermes session duration

11m 46s

API endpoint

http://127.0.0.1:1234/v1

Run Summary

Checkpoint	Status	Evidence
Confirm loaded model	Passed	LM Studio showed `qwen3.6-27b-mtp` loaded and idle with a `69000` token context before the run. Hermes was corrected to show the same model in the session header.
Set reasoning high	Passed	Hermes confirmed `Reasoning effort set to 'high' (saved to config)`.
Submit benchmark prompt	Passed	The prompt instructed Hermes to use `Qwen3.6-27B-MTP-IQ2_M`, read `/Users/armanshawon/Documents/Benchmark/test-prompt.md`, and create the required result file in the Benchmark folder.
Prompt processing	Very slow	LM Studio remained in `PROCESSINGPROMPT` for several minutes. Hermes still showed `0/69K` and no tool calls.
Generation	Unusable	LM Studio moved to `GENERATING`, but after more than 10 minutes of total request time there was still no usable Hermes-visible response or tool call.
Read `test-prompt.md`	Not reached	Hermes exited with `0 tool calls`; no file-read tool was called.
Create benchmark HTML	Not reached	No benchmark artifact was written by the tested model. The existing `hermes_local_model_agent_benchmark_submission.html` was an older file and was not produced by this run.

Timing

Metric	Observed value
Hermes session duration	11m 46s
Single API request elapsed time shown by Hermes	634.25s
Hermes request context	2 messages, about 8,668 tokens
Tool calls completed	0
Benchmark progress	No prompt file read; no task work started

Error Details

HTTP 400
Provider: custom
Model: qwen3.6-27b-mtp
Endpoint: http://127.0.0.1:1234/v1
Hermes error: Failed to load model "qwen3.6-27b-mtp". Error: Operation canceled.
LM Studio log: Model unloaded.

The final HTTP 400 happened after the active request was cancelled by unloading/ejecting the model. The benchmark decision was already a fail because the model produced no actionable response after more than ten minutes.

Timeline

Time	Event
2026-05-27 06:46:21 +0600	Fresh Hermes session opened. Session key: `20260527_064621_bfa0e3`.
Start of run	LM Studio confirmed `qwen3.6-27b-mtp`, size `12.87 GB`, context `69000`, parallel `4`, device `MGPC`.
Before benchmark prompt	Hermes confirmed `/reasoning high`.
Early run	LM Studio showed `PROCESSINGPROMPT`; Hermes showed `0/69K`.
Mid run	LM Studio switched to `GENERATING`, but Hermes still showed no model response and no tool calls.
2026-05-27 06:57:07 +0600	LM Studio logged `Model unloaded` after the model was ejected.
2026-05-27 06:57:13 +0600	Hermes wrote a debug dump at `/Users/armanshawon/.hermes/sessions/request_dump_20260527_064621_bfa0e3_20260527_065713_974966.json` and reported HTTP 400 / operation cancelled.
End of run	Hermes session exited after `11m 46s` with `1` user message and `0` tool calls.

Tool Call Detail

Tool or command	Purpose	Result
`hermes`	Started a fresh Hermes CLI session.	Succeeded; model header showed `qwen3.6-27b-mtp`.
`/reasoning high`	Set Hermes reasoning effort to high before the test prompt.	Succeeded; Hermes saved the setting.
Benchmark prompt	Requested the model-specific benchmark and asked Hermes to read `test-prompt.md`.	The model did not return a usable response before being failed for speed.
Hermes model tool calls	Expected file read and result-file creation actions.	None occurred. Hermes summary showed `0 tool calls`.

Judgment

Fail this run as unusably slow for the Hermes benchmark on this machine. The model did not reach the first required action after more than ten minutes, and the final cancellation occurred because the model was unloaded after the fail decision.