Make Sandbox Epoch a Fencing Token Before Free Compute Vanishes Mid-Write

Here is the event order I use as a paper counterexample when someone treats a free sandbox as durable storage. The planner sends apply_patch, the sandbox answers 200, and the saga steps forward as if the write were fenced. Then the replica is reclaimed, a retry lands on cold disk, and the file is gone. Did the tool succeed? The HTTP log says yes. The filesystem says no.

Would you let the planner commit that step? I would not, not on a replica that can vanish without a fencing token. Free compute is useful, but it is a preemptable replica, not a disk you own. This article is an architecture review of that constraint, not a production war story and not a benchmark.

The invariant the common sketch fails

The common sketch stores tool_status = success as soon as the sandbox returns a 200. That sketch assumes the replica that accepted the write is the replica that will still exist on the next tick. Free servers break that assumption in the most ordinary way: they go away.

Invariant. A planner must not commit a mutating tool step until a write receipt joins a live sandbox epoch, and that epoch is still the fencing token for the replica. If the epoch has moved, the system must reject or compensate. Replay without a new fence is how you mint a second write.

That is a protocol question, not a model-quality question. If your eval harness treats a vanished sandbox as a flaky model, you will debug the wrong layer.

Declared assumptions

I am reviewing a candidate design on paper, with these assumptions held fixed.

1. The planner is the only component allowed to commit saga state.
2. The sandbox is a remote workspace that may be preempted between any two messages.
3. Mutating tools (write_file, apply_patch, run_migration) require a fenced write; read-only tools do not.
4. Duplicate delivery is possible after a timeout, an ack loss, or a reconnect.
5. We do not assume dedicated hardware, reserved capacity, or a measured SLA.
6. The fixture below is a proposed simulator, not an executed production test.

If your sandbox is a reserved VM with a disk you control, this review is heavier than you need. If your sandbox is a free replica, this is the failure class you should canary first.

Constraints and data flow

The design has three objects, and only one of them is allowed to tell the truth about commits.

• Planner log. Append-only saga steps. A step is pending, fenced, committed, or compensated.
• Epoch lease. A monotonic integer owned by the current sandbox generation. Heartbeats keep it live.
• Replica disk. Ephemeral. Writes are visible only while the epoch that accepted them is still live.

Data flow is narrow on purpose. The planner mints a step id, the sandbox returns {epoch, receipt}, and the planner commits only on a join of those two values. No other signal, including a 200 with an empty body, is allowed to move the saga.

planner --(step_id, tool, epoch?)--> sandbox
sandbox --(epoch, receipt | preempted)--> planner
planner --(commit | reject | compensate)--> saga log

Enter fullscreen mode Exit fullscreen mode

Notice what is missing. There is no path from “the model sounded confident” into the commit bit. There is also no path from “the HTTP client did not throw” into the commit bit. Those are observations, not fences.

Failure domains

I split the system into four domains because they fail independently, and mixing them hides the counterexample.

Domain What it owns How it fails Planner process Saga log, step ids Crash after receipt, before commit Network In-flight apply and heartbeat Loss, reorder, duplicate Sandbox replica Epoch, ephemeral disk Preempt, cold start, silent drop Model loop Tool arguments Retry with the same step id

The interesting collision is replica preemption plus a late 200. The planner sees success from a generation that no longer exists. If you commit there, you have advanced past a write that the next replica cannot see. That is not eventual consistency. That is a lost update with a green dashboard.

Can the model loop save you? No. A retry that does not carry the new epoch will either no-op on a dead generation or double-apply on a live one. The fence has to sit under the tool, not inside the prompt.

Sequence model

I keep a tiny state machine on paper. If you cannot draw the join, you cannot test it.

sequenceDiagram
    participant P as Planner
    participant L as Epoch lease
    participant S as Sandbox
    P->>L: acquire(step_id)
    L-->>P: epoch=7
    P->>S: apply_patch(step_id, epoch=7)
    Note over S: replica preempted, epoch becomes 8
    S-->>P: 200 receipt epoch=7 (late)
    P->>L: join(receipt.epoch, live_epoch)
    L-->>P: mismatch, reject
    P->>P: compensate or re-acquire

The late 200 is the violating event. A common implementation stores success on that 200 and never asks the lease. The invariant dies on that single message.

Build the fencing protocol in numbered steps

This is the workflow I want a reviewer to implement against a local simulator first. Remote wiring comes after the properties pass.

1. Mint a step id in the planner, not in the sandbox. The replica does not own identity, because it can die.
2. Acquire an epoch before any mutating tool. Treat the epoch as a fencing token, not as a cache key.
3. Send the epoch with the write. A sandbox that sees a stale epoch must refuse the body, even if the path is valid.
4. Heartbeat the epoch on a shorter period than your preemption budget. A missed heartbeat is preemption, not a slow model.
5. Join receipt and live epoch before committed. A 200 with a stale epoch is a reject, not a retry hint.
6. Compensate on mismatch when the tool had side effects outside the replica. Replay only after a fresh acquire.
7. Record the denominator. Count mutating_steps, not wall time, when you say the protocol held.

If step four feels operational, ask the question a different way. Are you measuring liveness of the replica, or are you hoping the next tool call will notice the disk is empty? Hope is not a fence.

Minimal simulator

The fixture is deliberately small. It is meant to be copied into a unit test, not wrapped in a framework. Label it as unexecuted until you run it on your machine.

from dataclasses import dataclass

@dataclass
class Replica:
    epoch: int = 0
    live: bool = False
    disk: dict = None

    def __post_init__(self):
        self.disk = {}

    def acquire(self):
        self.epoch += 1
        self.live = True
        self.disk = {}
        return self.epoch

    def preempt(self):
        self.live = False
        self.disk = {}
        # Epoch advances so a late receipt cannot fence a new generation.
        self.epoch += 1

    def write(self, step_id, epoch, path, body):
        if not self.live or epoch != self.epoch:
            return {"ok": False, "reason": "stale_epoch", "epoch": self.epoch}
        self.disk[path] = body
        return {"ok": True, "receipt": f"{step_id}:{epoch}", "epoch": epoch}


class Planner:
    def __init__(self, replica):
        self.replica = replica
        self.log = []  # (step_id, state, epoch)

    def mutating_step(self, step_id, path, body, preempt_before_join=False):
        epoch = self.replica.acquire()
        self.log.append((step_id, "pending", epoch))
        ack = self.replica.write(step_id, epoch, path, body)
        if preempt_before_join:
            self.replica.preempt()
            # Late 200 still in hand. Join must fail.
        live = self.replica.live and ack.get("epoch") == epoch and ack.get("ok")
        if not live:
            self.log.append((step_id, "rejected", epoch))
            return "reject"
        self.log.append((step_id, "committed", epoch))
        return "commit"


def test_late_receipt_cannot_commit_after_preempt():
    p = Planner(Replica())
    assert p.mutating_step("s1", "/app.py", "x", preempt_before_join=True) == "reject"
    assert ("s1", "committed", 1) not in p.log


def test_live_epoch_may_commit():
    p = Planner(Replica())
    assert p.mutating_step("s2", "/app.py", "x") == "commit"
    assert p.replica.disk["/app.py"] == "x"

Enter fullscreen mode Exit fullscreen mode

Run it as a module and keep the assertion names as the acceptance rule. I care about one denominator: mutating steps that reached committed without a live matching epoch. That count must stay zero.

python -m pytest -q sandbox_epoch_fence.py

Enter fullscreen mode Exit fullscreen mode

Inject three failure classes before you call the protocol done: preemption before write, preemption after write but before join, and duplicate apply with a stale epoch. If any of those three can mint a committed row, the design is not ready for a remote replica.

Tradeoffs

Choice What you gain What you pay Commit on HTTP 200 Low latency, simple client Lost writes after preemption Epoch fence + join Rejects late receipts Extra round trip, more rejects Always replay the tool Recovers empty disks Duplicate side effects Compensate on mismatch Honest saga state You must write compensations Dedicated replica Fewer preemptions Cost, and you still need a fence

I would not pick the first row for any mutating tool, even on a quiet afternoon. Latency you cannot attribute to a live epoch is not latency you can spend. Throughput without a denominator is a story, not a model.

Where a free remote server actually fits

You still need a replica that is allowed to vanish, because that is the failure class the simulator is canarying. A reserved box will under-inject preemption and give you a false pass. This is the one place I would use a free remote sandbox on purpose.

MonkeyCode is an open-source project with free model access and a free server option, which matches that preemptable-replica role. Disclosure: This article was prepared as part of MonkeyCode’s product outreach. I am not claiming a quota, a model list, a hardware SKU, or a duration. I am claiming a failure domain: a workspace you do not own can disappear between apply and join.

Wire the same epoch header you used locally. If the remote process dies, treat the next connect as preempt(), not as a retry of the old generation. If you cannot attach an epoch to the workspace, you do not yet have a fence. You have a hope that the next prompt will notice.

If you try that path, keep promotion out of the acceptance rule. The rule is still zero committed steps with a stale epoch, measured over mutating steps, with preemption injected on purpose.

Limitations, and who should not use this

This review is for planners that mutate a remote workspace and then continue a saga. It is not a general reliability sermon.

Do not use a free replica as the system of record for billing, medical state, or anything you cannot compensate. Do not skip the fence because the model is “usually right.” Do not treat this simulator as a load test. It has no arrival process, no tail latency, and no claim about tokens.

Skip this design if every tool is read-only, or if the sandbox is a container your orchestrator fences with its own generation number. In that case you already have an epoch. You should still join it. You should not invent a second one.

What I would change next

The current fence is a single integer. That is enough to kill the late-200 counterexample, and it is not enough to prove a prefix after a replica revives with leftover disk. Next I would pair epoch with a log index, so a revived sandbox can say “I still have writes 1..k under epoch 7” instead of wiping on every acquire.

I would also split reject from compensate in the planner. Reject is the right answer when the write never became visible. Compensate is the right answer when a side effect may have escaped the replica, such as an email or a payment intent. Blending those two into a blind replay is how you get the duplicate-apply bug the fence was meant to stop.

Finally, I would property-test event order instead of listing three cases. Generate shuffles of {acquire, write, preempt, late_receipt, retry} and assert the invariant on every shuffle. Three unit tests are a tutorial. The shuffle is the review.

Acceptance rule

Declare the load as N mutating steps with preemption injected on at least one third of them. The protocol holds only if committed steps with a mismatched epoch equal zero, and if every rejected step is either retried under a new epoch or compensated. If you cannot state that sentence with a denominator, you are not ready to scale the planner.

Which event order still breaks the invariant in your sketch, and should the system reject, replay, or compensate? If the answer is “the 200 was green, so commit,” the replica is already gone, and the saga has moved without a fence.