California – October 31, 2026 -- Switching off Gemma 4's default reasoning with a single flag raised accuracy by up to 48 points, cut cost 6x to 8x and made a correct answer up to 19x faster on short verifiable tasks, according to a 27-model local benchmark run on a Strix Halo machine with a power meter attached. The 12B and 26B variants rose from about 52% to 100% on the easy suite.
Each model answered 190 checkable tasks twice, under two different prompts. Electricity was priced at California summer rates of 34 cents per kilowatt-hour off-peak and 58 cents on-peak.
Gemma 4 reasons on 95 of 95 tasks even when told not to
Asked for a one-line answer with no explanation, Gemma 4 emitted reasoning on every task in both prompt conditions. On a 4,096-token budget, the 12B and 26B spent the entire allowance thinking and returned nothing on roughly half the questions.
Both --reasoning off in llama.cpp and enable_thinking: false in the chat template worked on the tester's build. An open upstream issue reports the opposite, and the Gemma 4 reports inside it contradict each other. The 31B gained only 15 points because it was less budget-starved to begin with.
The tester warns that --reasoning-budget 0 is not equivalent. It empties the API's reasoning field while the model keeps reasoning in the ordinary content field, so tools counting reasoning tokens report silence.
Reasoning still wins on hard work, with caveats on the data
Google publishes GPQA Diamond scores of 78.8, 82.3 and 84.3 for the three Gemma 4 models. With reasoning off, the tester measured 61, 67 and 70. Part of that gap stems from Q4_K_M weights and a 100-question subset, and GPQA was never run with reasoning on to separate the effects. The default suits genuinely hard work; on short verifiable tasks it burns budget on problems that were not hard.
Permission to think lifts instruct models 22 to 41 points
Replacing the "answer immediately" instruction with a free-form prompt raised accuracy by 22 to 41 points while token counts stayed in the low hundreds. qwen3-30b-a3b-instruct went from a coin flip to 98%.
Models trained to think cost far more. qwen3-30b-a3b-thinking costs 5.8x its instruct sibling per right answer, and qwen3-4b-thinking 6.4x. The extra spend returned nine accuracy points at 4B and minus three at 30B.
The invitation can backfire. trinity-nano gained nine points but went from 61 to 1,181 tokens per task, a 19x spend. phi-4-reasoning lost 26 points after the free-form prompt pushed it past the token cap on tasks it had answered correctly when forced to be brief.
Twelve of 27 models ignore the terse prompt, and a 200x cost gap follows
Twelve models emitted reasoning despite the one-line instruction, and 11 did so on all 95 tasks. phi-4-reasoning was not counted among them because it streams thinking as ordinary output rather than into the API's reasoning field. It wrote 2,811 tokens in reply to "one line, no explanation."
Obedient models llama4-scout, granite4-h-small, phi-4 and qwen3-next-80b-a3b answered in 14 to 15 tokens. The gpt-oss pair and nemotron emit reasoning but spend only a couple of hundred tokens on it.
Truncation and spiraling turn tokens into pure cost
Every model in the bottom third of the cost table emits more than 1,000 tokens per correct answer, and several exceed 2,000. phi-4-reasoning hit the 4,096-token cap on 78% of free-mode tasks, which is most of why it costs $5.57 per thousand answers.
trinity-nano scored zero of 14 on constraint puzzles under free reasoning, with 12 of 14 running to the cap and averaging 3,574 tokens. Told to answer immediately, it averaged 10 tokens and got three right.
Quadrupling the token cap added 2.5x to 9x cost and no accuracy
Re-running phi-4-reasoning and both Arcee models with 4x the room gained nothing. trinity-nano still hit the larger cap on 22% of tasks instead of 25% and picked up four tasks, within run-to-run noise.
For trinity-mini, the tight budget delivered the same accuracy for a third of the energy. It scores 91.6% when told to be brief and 90.5% when told it can think. trinity-nano has the lowest per-token cost measured at 0.33 J, and trinity-mini ranks fifth of 27. The tester notes trinity-nano is a preview release.
A llama.cpp setting nearly corrupted the larger-budget test. A request for 16,384 tokens returned 7,903 because the server's -c context is divided across --parallel slots: a 32,768-token context with four slots gives each request 8,192.