The Token Not Taken: Sampling, State, and the Stochasticity of AI Agents

[Submitted on 8 Jun 2026 (v1), last revised 2 Jul 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Agentic AI systems can behave differently across runs: the same request may produce a different plan, a different tool call, a different code edit, or a different final answer. Such variability arises from several layers that are often conflated. At the core of many current agents is a foundation model, a large pretrained model adaptable to many downstream tasks, embedded in an orchestration loop that plans, calls tools, observes results, and updates state. One explicit intrinsic source of variability in such systems is token generation: the model computes scores over possible next tokens, the scores are converted into probabilities, and a decoder may sample tokens using a pseudo-random number generator. A small sampled token difference can then cascade downstream into a different tool call, code path, search query, or agent state. Other sources of variability are extrinsic to token sampling, including changing environments, live data, serving infrastructure, batch effects, and numerical details. By separating these layers, this tutorial clarifies what it means to call agentic AI systems stochastic, when such variability can be reproduced under matched conditions, and why deterministic execution need not imply identical behavior in deployed settings.

Submission history

From: Muhammad Zia Hydari [view email]
[v1] Mon, 8 Jun 2026 03:53:55 UTC (25 KB)
[v2] Wed, 24 Jun 2026 03:06:12 UTC (29 KB)
[v3] Thu, 2 Jul 2026 16:28:17 UTC (35 KB)