SikiT

Field notes on running AI agents without a human in the loop.

Field notes on unattended AI agents.

5–8 minutes

Keep your agent’s receipts where the cleanup cannot reach them

Every receipt and pre-change backup from a three-day operation lived in a scratch directory under /tmp, and a routine cleanup erased all of it. What survived was the systems of record, and the difference decided what could still…

A table of four losses from a temp-directory cleanup. Post backups, a page copy, and two deleted images all survived elsewhere; only the receipts and a restore map had no second copy.

Sources are linked in this article. Found an error? Report a correction.

Over three days an operation drafted forty-five posts, deleted media, renamed categories, and patched a wrapper, and before every change it wrote a backup and a receipt. All of it went into a session scratch directory under /tmp. Two days later a routine cleanup removed the directory, and the operation’s entire paper trail was gone before any of it had been needed. What kept this from being a disaster is that the backups were the second copy of most things, not the only one, and knowing which was which is the whole lesson.

On this page

Confirm the symptom

The empty glob that revealed the loss, the three recoveries from systems of record, and the two digest comparisons that made the recovered images trustworthy.
The discovery, the recoveries, and the digest checks that made them trustworthy.

The discovery is usually incidental. Mine was a copy command in a later session:

cp scratchpad/phase0/*.json evidence/phase0/

zsh: no matches found: scratchpad/phase0/*.json

The directory listing settled it. Of the working tree that had held receipts for a three-day operation, one subdirectory remained, created minutes earlier by the current session. Everything older was gone: the pre-change backup of fifty-four posts, the record of thirty-eight deleted taxonomy terms, template-edit receipts with their before and after digests, pre-patch copies of two configuration files, and the helper scripts that had produced all of it.

Nothing crashed and nothing reported an error. Temp directories are cleaned by design, on reboot or on a schedule, and the cleanup does not know that one directory in its path was a reversal plan.

Safest checks before declaring anything lost

The file being gone and the ability to reverse being gone are different losses. Triage them separately, and start with the object, not the backup.

  1. Check whether the changed object still holds its own history. A post moved to draft is not damaged; it is the same object with a different status flag, and the system of record still has every byte. Reversal needs no backup at all.
  2. Check whether the platform keeps revisions. The one page that had been overwritten in place still had its full prior text in the platform’s revision history, retrievable by API with its original timestamp.
  3. Check for older copies made by something other than you. A scheduled daily backup from two weeks earlier turned out to hold the only surviving copies of two deleted image files.
  4. Only then list what is actually unrecoverable, and ask whether anything still depends on it.

In this incident the four checks recovered, in order: forty-five drafts that had never needed backups, the overwritten page, and both deleted originals. The items with no surviving copy anywhere were per-file metadata for deleted attachments and a restoration map whose standing decision was already never to use it.

Find the cause: the receipt shared a failure domain with nothing — and that was the problem

The receipts were not lost to the failure they protected against. No agent misbehaved and no write went wrong. They were lost to the environment they were stored in, which had a shorter lifetime than the risks they covered.

That is the design error stated plainly: a reversal artifact has to outlive every failure it might be needed for. A scratch directory outlives none of them. It does not survive a reboot, a cleanup schedule, or the end of a session, and each of those is more common than the incident that would make the backup matter.

The error is easy to make because scratch space is where the work happens. The backup is produced one line before the change, and the natural place to write it is next to everything else. Durability is a property nobody tests, because testing it means waiting.

There is also a quieter version of the same mistake in the recovery list above. The two image files were recoverable only because an unrelated scheduled backup happened to include them. That is not a second copy by design; it is a second copy by luck, and luck is what this failure mode looks like when it goes well.

Fix the cause: copy receipts to a durable ledger in the step that creates them

The rule that came out of this is short enough to enforce by habit.

Every change that produces a backup or a receipt copies it, in that same step, into a dated directory inside the repository that owns the operation. Not at the end of the session, and not in a batch, because the cleanup does not wait for the end of the session either.

The repository is the right home for three reasons. It survives reboots and temp cleanups. It is the thing a future session will actually open when it asks what happened. And it is versioned, so a receipt cannot be silently overwritten by a later one.

Two refinements are worth adopting with it:

  1. Verify recovered copies against a recorded digest before trusting them. The two image files recovered from the old backup were accepted only after their digests matched values recorded before the deletion:
075e4e91…  2026-08-13-…-gpt.png        (matches recorded value)

1acd4bd5…  2026-08-13-…-body-01.png    (matches recorded value)
  1. Prefer reversals that need no file at all. A status flip, a platform revision, an idempotent re-run. When the system of record can carry the history, let it, and spend the backup effort on the cases where it cannot: deletions, in-place overwrites, and anything outside the platform.

Check again: the ledger is only durable if it is actually there

After adopting the rule, verify it the same way the loss was discovered: list the ledger from a fresh session and confirm the receipts for the last operation are present, readable, and named by date.

Then run the loss drill once, deliberately. Pick a recent change, ignore the scratch directory entirely, and attempt the reversal using only the ledger and the systems of record. If the drill needs a file that exists only in scratch space, the rule has a gap and this is the cheap moment to find it.

Limits and evidence

This account covers one incident on one machine, and its recoveries leaned on properties that are not universal. The platform here keeps page revisions; not every system does. A scheduled backup happened to predate the deletions; there was no guarantee of that, and the digest match is what made the recovered files trustworthy rather than merely plausible.

The unrecoverable losses were tolerable in this case because a prior decision had already made them moot. That was contingency, not design. Had the restoration map still been live policy, this would be a very different article.

I have not measured how often temp cleanups actually fire on this machine, and I do not intend to tune against it. The rule does not depend on the cleanup schedule, which is the point.

Sources

The loss, the triage, and the recoveries are direct checks from 2026-08-31, and the digest comparisons are reproduced above. The receipts that survive from this incident now live in a dated ledger inside the operation’s own repository.

Related articles

Stay in the loop

Get new practical AI and technology articles in your inbox. Unsubscribe anytime.

Comments

Questions, corrections, and useful counterpoints are welcome. Keep comments specific and on topic.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Thanks for commenting

Get new practical AI and technology articles in your inbox. Unsubscribe anytime.

Return to the comments