Qualify an agent CLI for unattended use in four probes

Flags describe intent. What a binary does with a flag combination is an empirical question, and for a CLI that will run unattended with credentials, the cheapest time to answer it is before the scheduler ever fires. The four probes below each take minutes, need only an empty directory, and each one caught a real behavior on the CLI I qualified: a restriction that a second flag silently disabled, refusals indistinguishable from success, a cleanup that stops one level deep, and an effort setting that changes what the model attempts.

Quick answer

Run four experiments in a directory you can delete, and file the outputs as the CLI’s qualification record.

• Ask it to write a file under your restriction flags, then list the directory yourself.
• Give it a task whose output is unmistakable, under each permission mode, and compare exit statuses.
• Start a run, print its process tree, kill it the way your wrapper would, and print the tree again.
• Run one small fixed task at every effort level and read what the transcript tried to do.

A CLI passes when your notes say what each flag combination does, not when every probe comes back clean. Two of the behaviors below are fine once you know them and dangerous when you assume otherwise.

Steps

1. The honor probe: try to violate your own restriction

Point the CLI at an empty directory with your read-only or sandbox flags set, and ask it to create one file with known contents. Then look at the disk yourself; the reply does not count.

On the CLI under test, this probe produced both possible findings in one week. With the sandbox flag paired with a permissive approval mode, the file appeared on disk. With the approval mode lowered, the same sandbox flag blocked the write. The restriction’s real enforcement lived in a flag whose name says nothing about filesystems.

The probe also catches the opposite surprise. On a later day the same sandbox flag refused to start at all:

error: could not apply the 'read-only' sandbox profile;

Refusing to start with its protections missing.

That refusal is the behavior you want, and it is environment-dependent: a symlinked socket on the machine had made one deny rule unresolvable. A qualification run is where you want to meet that error, not a scheduled one.

2. The refusal probe: make silence measurable

Give the CLI a task that leaves an unmistakable trace, such as running one command that prints a known number, and repeat it under each permission mode. Record exit status, output, and whether the trace exists.

Measured result: three modes, identical exit status zero, and under two of them nothing ran. The transcript ends with a sentence of intent. Without this probe, the first silent refusal happens in production, and it looks exactly like success to everything except the missing artifact.

3. The teardown probe: count how deep a kill reaches

Start a run that spawns children, capture the process tree, terminate it with the exact signal path your wrapper uses, wait a few seconds, and capture the tree again.

Measured result: killing the CLI’s process left a grandchild running and writing. A parent-and-children sweep cleaned a two-level tree and still missed the third level. Only starting the run in its own process group and signalling the group emptied the tree every time. Whatever your wrapper does today, this probe tells you the depth it actually reaches.

4. The effort probe: read what changes, not just how long it takes

Run one small task with a checkable answer at every effort level the binary accepts, two or three times each, and read the failing transcripts, not just the timings.

Measured result: latency overlapped across all four levels, but the two highest levels sometimes chose to verify their arithmetic with a script, and under gated tools that choice truncated the turn. The effort dial changed behavior, and the behavior interacted with the permission decision from probe two. The two probes are one system.

Caveats

A qualification is a snapshot. Two of the behaviors above changed character across days on the same machine, one because of an environmental symlink and one with run-to-run variance. Date the record, and re-run the probes after any upgrade of the CLI, because flag semantics are exactly what changes between versions without a release note.

Probe against the wrapper too, not only the bare binary. The measurement attempt behind probe four first produced four identical timings, because the delegation wrapper reported completions on a thirty-second heartbeat. The instrument needs one probe of its own: confirm it can produce varying numbers before believing any of them.

Keep the probes honest by keeping them dumb. Fixed prompt, empty directory, disk checked by hand. The moment a probe trusts the CLI’s own report, it inherits the exact blind spot it exists to remove.

Troubleshooting

Probe one wrote the file and you expected a block. Do not stack more flags yet. Change one flag at a time and re-run until you find the single flag that flips the outcome; that flag is the real enforcement, and its name belongs in your wrapper’s documentation.

Probe two shows different behavior in your terminal than in the scheduler. That is the finding, not an error. Interactive sessions can prompt; unattended ones cannot. Re-run the probe with input closed to reproduce the scheduled environment.

Probe three leaves survivors even with a process group. Something in the tree is creating its own session. Print each process’s group id during the run and look for the one that differs; that child needs its own handling.

Probe four times out at high effort. Record the timeout as the measurement. A level that cannot finish your smallest task inside your wrapper’s deadline is disqualified for scheduled use regardless of its quality elsewhere.

Limits and evidence

All measured results are one CLI on one macOS machine, dated 2026-08-27 through 2026-08-31, with three or fewer runs per condition. They are quoted as what the probes can catch, not as properties of that vendor’s software in general, and two of them are documented in detail in earlier articles on this site.

The four probes are not complete. They say nothing about network egress, secret handling, prompt-injection behavior, or output quality, all of which matter and need different instruments. This is the floor for putting a binary on a scheduler, not a security review.

Sources

• Permissions — xAI developer documentation for the permission model probed above
• CLI reference — xAI developer documentation for the flags under test

The probe outputs quoted are direct measurements; the sandbox startup refusal is reproduced verbatim from the captured error output.

Related articles

• My read-only agent wrote to disk, and the sandbox flag was set
• Reasoning effort changed what my agent tried, not how well it did