Reasoning effort changed what my agent tried, not how well it did

Effort levels read like a quality dial: pay more latency, get better answers. I timed one deliberately small task across all four levels of one agent CLI, three runs each, and the dial did not behave like that at all. Latency overlapped across every level. Correct answers came from every level. The only variable that moved with effort was what the agent chose to attempt, and on the two highest settings that choice collided with the sandbox and produced the worst possible outcome for automation: a clean exit with no answer.

Thesis: effort is a behavior dial, and behavior meets the environment

Raising reasoning effort on a small task did not buy speed or accuracy. It raised the model’s appetite for verifying its own work with tools, and whether that appetite becomes extra rigor or silent failure is decided by the permission configuration, not by the model. Choosing an effort level for unattended runs is therefore not a model decision. It is a joint decision with the environment the run executes in.

Evidence: twelve runs, one task, four levels

The task was chosen so correctness is checkable at a glance: sum the integers from 1 to 400 divisible by 3 or 5 but not by 15, reply with the number only. The answer is 32,403. Runs used a read-only sandbox, the default permission mode, web search off, and the levels the binary itself enumerates when handed an invalid value:

--effort/--reasoning-effort: unknown effort level 'bogus';

use one of: xhigh, high, medium, low

Latency does not order the levels. The slowest run in the table is a medium and a high, the fastest is a low, and every level lands in the same 13-to-35-second band.

The failures are the finding. None of the four failed runs produced a wrong number. Each produced a sentence and stopped:

high, run 1:  "...inclusion-exclusion and then verify it

               with a direct calculation."          exit 0
xhigh, run 2: "...inclusion-exclusion, then double-check
               with a short script."                exit 0

At low and medium, the model answered directly, six for six. At high and xhigh it sometimes planned a tool-assisted verification, and in a non-interactive run under the default permission mode that tool call has nowhere to go. The turn ends after the statement of intent, with a clean exit status. The extra effort was spent deciding to be more careful, and the environment converted that care into nothing.

The harness lied first

The first version of this measurement produced four numbers that were all exactly 30.0 to 30.1 seconds, across levels whose direct times vary by a factor of two. The delegation wrapper used for the first attempt polls its child on a heartbeat whose default is thirty seconds, so every completion was reported on the next heartbeat boundary. The uniformity of the numbers was the tell: real workloads vary, and a measurement that does not vary is measuring the instrument. All times in the table above come from timing the CLI directly.

Counterargument: one small task cannot condemn a dial

Three runs per level is an existence proof, not a rate. The honest claims are narrow: at least one high run and one xhigh run truncated, and no level separated on latency for this task.

The task is also deliberately trivial, and that cuts both ways. A problem this small gives higher effort nothing to convert into accuracy, so the absence of a quality gain here says little about hard tasks. It is precisely the small-task regime, though, that scheduled automation lives in most of the day, and that is the regime where the extra behavior showed up as pure downside.

The truncation itself is also environment-specific. In an interactive terminal the same impulse would surface as a permission prompt and a human would approve it. Nothing here says high effort is broken; it says high effort plus gated tools plus nobody watching is a specific combination with a specific failure, and it is the combination automation runs in.

Implications: pick the pair, not the level

Three practical rules fall out of the measurements.

Choose effort and permissions together. If tools are blocked, high effort buys latency variance and a new failure mode while low effort answered everything this task asked. If tools are allowed, the verify impulse at high effort is exactly the behavior you want from an unattended agent, and the cost calculus flips.

Detect the truncation, because the exit status will not. A clean exit with an intent sentence and no artifact is the same shape measured before on this site with permission modes, and the same defense applies: require an artifact the caller can check, and treat its absence as failure regardless of status.

Verify the measuring stick before the measurement. The thirty-second heartbeat quantization would have flattened any real difference between levels into identical numbers. Any harness that reports timings should first be shown to produce varying ones.

Sources

• CLI reference — xAI developer documentation for the CLI under test

The effort levels are enumerated by the binary’s own validation error, quoted above. The twelve timed runs, the two truncated transcripts, and the four wrapper-quantized timings are direct measurements from 2026-08-31 on macOS, and the task’s expected answer is checkable by hand.

Related articles

• Exit code 0 from an agent that did nothing
• Qwen 3.8 27B thinks by default, and the default effort is xhigh