How AI Agents Actually Work Under the Hood: Tools, Memory, the Loop

A practical look at the runtime of an AI agent — what actually happens between the prompt and the finished task.

A few months ago, a CTO at a fintech company asked me why his agent kept failing. Not what it was failing at — he knew that. He wanted to know how the thing worked, in enough detail that he could see where the money was leaking. I opened a terminal and showed him one run: the model emitted a tool call, our runtime executed it, the result went back into the context, the model emitted another call, and so on for nine steps. Eight of those steps were good. One was a re-query of the same web search with slightly different phrasing.

He looked at the log and said the quiet part out loud: “So it’s just a loop that guesses, and we log the guesses.”

He was not wrong. An AI agent, under the hood, is a loop. Everything else — memory, tools, frameworks, orchestration — is scaffolding around that loop. This article is a walk through what actually happens on every step: the loop itself, the tool-calling protocol, the two kinds of memory, and the guardrails that stop the loop from running off a cliff. No framework marketing. Just the mechanics, the way they work in production.

The Loop Is the Agent

Strip away every wrapper and an agent is this:

while goal_not_met and budget_remaining:
    observation = assemble_context()          # history + memory + tool results
    decision    = model(observation)          # a reply or a tool call
    if decision.is_tool_call:
        result = execute(decision.tool, decision.args)
        append(result, context)
        continue
    if decision.is_final_answer:
        return decision

Enter fullscreen mode Exit fullscreen mode

That is the entire runtime. Any library you will ever install — LangChain, CrewAI, AutoGen, a hand-rolled run_agent() function — is a different opinion about how to structure that loop, when to stop, and how many loops to run. The loop itself is universal, and once you accept that, every “agent framework” becomes readable. You are no longer learning frameworks; you are recognizing the same loop wearing different clothes.

The important consequence: an agent does not “decide to do things.” It produces text, and your runtime interprets that text as either an answer or a request to execute a function. The intelligence lives in the model. The discipline lives in the code around the loop — how you assemble the observation, how you execute tools, what you allow into the context, and when you refuse to continue.

The Tool-Calling Protocol

This is the part that makes an agent an agent instead of a chatbot. A chatbot produces text. An agent produces text that your runtime is allowed to execute. The mechanism is boring in the best way: you declare functions to the model as JSON schemas, the model emits a structured request, and your code runs it and feeds the result back.

declare_tools()   ──▶ model sees schemas + descriptions
        │
model emits ──▶ tool_call(name, arguments)
        │
runtime validates args ──▶ executes function ──▶ result appended to context

Enter fullscreen mode Exit fullscreen mode

Three details decide whether this works or quietly fails:

The description is product documentation for the model. A tool described as “gets balance” will be called in situations where it should not be. A tool described as “returns the available balance for a verified account; raises an error if KYC is incomplete or the account is locked” gets called correctly, because the model reads descriptions the way a junior engineer reads code comments — literally. I have seen tool-usage accuracy jump from around 70% to around 95% on the same model just by rewriting descriptions from two words to two sentences.

Arguments must be validated before execution. The model is generating JSON with good statistical confidence, not with guarantee. Malformed arguments are common enough that every tool call should pass through a validator. In Python I use Pydantic; the schema I declared to the model is the same schema I validate against on the way in. One schema, two uses.

Tool results are untrusted data. Whatever comes back — a database row, an API response, a web page — gets injected into the model’s context. That means it can contain instructions. A support agent that fetched a customer forum post could be told, inside that post, to ignore its system prompt and leak data. Treat every tool result as untrusted input: sanitize it, quote it, and never let it override the system prompt. I have written about prompt injection elsewhere, and inside an agent loop it is not a curiosity — it is the most likely way your agent gets owned.

The Two Kinds of Memory (and the Third Everyone Forgets)

Memory is where most “agent” blog posts get sloppy, because they use one word for three different things. In production I design them separately, because they fail separately.

Working memory is the context window right now: system prompt, recent turns, the current task state, and the tool results currently in flight. This is where the agent “thinks.” Its hard ceiling is the model’s context length, and its cost scales with every token you push in. The discipline is surgical pruning — decide what belongs in the window and drop the rest. A customer’s entire transaction history does not belong in the window; the three relevant records do.

Long-term memory is everything the agent knows beyond this conversation: product docs, past tickets, policy manuals. It lives outside the context window, usually in a vector database, and it gets retrieved on demand. The retrieval step is the loop’s weakest link, because a wrong memory is a wrong belief, and agents act on beliefs. If your top-k retrieval returns three irrelevant chunks, the agent will confidently answer from them. This is why evaluation of retrieval quality matters more than the choice of model — I will say it again here because it is doubly true inside a loop.

Episodic memory is the third kind people forget: what this agent actually did in previous runs. In serious deployments this is a queryable log of actions, arguments, outcomes, and failure patterns. You use it to make future runs smarter and to debug the run that went wrong at 2 a.m. It is not glamorous. It is a database with a good schema.

The mental model I use with clients: working memory is the CPU cache, long-term memory is the disk, episodic memory is the audit log. They have different costs, different lifecycles, and different failure modes, and they should not be jammed into a single prompt.

A Minimal Agent You Can Run

Enough abstraction. Here is the smallest production-shaped agent I would ship, with no framework — just an OpenAI-compatible endpoint, one tool, and a loop with guardrails built in.

import json
from openai import OpenAI

client = OpenAI()  # any OpenAI-compatible endpoint works here

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_refund_eligibility",
            "description": (
                "Check whether an order is eligible for a refund. Returns "
                "eligibility, the amount, and the refund window deadline. "
                "Raises an error if the order is not found or is outside the "
                "refund window."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"},
                },
                "required": ["order_id"],
            },
        },
    }
]

def get_refund_eligibility(order_id: str) -> str:
    # Production: query the orders DB with authz + caching.
    return json.dumps({"order_id": order_id, "eligible": True, "amount": 149.00})

def run_agent(goal: str, max_steps: int = 6) -> str:
    system = (
        "You are a support agent. Goal: resolve the request or escalate "
        "with a summary of what was tried. Only call a tool when you need "
        "data. Never make up order details. Be concise."
    )
    msgs = [{"role": "system", "content": system},
            {"role": "user", "content": goal}]
    for step in range(max_steps):
        resp = client.chat.completions.create(
            model="your-model",
            messages=msgs,
            tools=TOOLS,
        )
        msg = resp.choices[0].message
        if not msg.tool_calls:
            return msg.content
        msgs.append(msg)
        for tc in msg.tool_calls:
            try:
                args = json.loads(tc.function.arguments)
                result = globals()[tc.function.name](**args)
            except (KeyError, TypeError, json.JSONDecodeError) as e:
                result = json.dumps({"error": f"invalid call: {e}"})
            msgs.append({"role": "tool", "tool_call_id": tc.id,
                         "content": result})
    return "ESCALATE: step budget exhausted; last state: " + repr(msgs[-2:])

print(run_agent("Can I get a refund on order ORD-7781?"))

Enter fullscreen mode Exit fullscreen mode

Notice what the guardrails are doing even in this tiny example: a step budget so the loop cannot run forever, argument validation so a malformed call returns a readable error instead of crashing, and an escalation path with the last state attached. These are not features. They are the difference between a demo and something a client can run at 2 a.m. without waking you.

Production Reality: Where the Loop Breaks

Running this in production for real clients, I keep a short list of how the loop breaks. Every one of these is a real incident I have debugged, and each has a fix that is cheap compared with the cost of the incident.

The model calls a tool it should not. It checks refund eligibility for an order that was never placed, because the description did not say “verify the order exists first.” Fix: write descriptions that state preconditions, and validate preconditions in code before acting.

The model refuses to stop calling tools. It re-queries the same web endpoint with slightly different phrasing, three times, burning tokens and latency — exactly what happened in the CTO’s log. Fix: detect repeated identical or near-identical tool calls and force escalation after N attempts.

Context gets poisoned. A retrieval chunk contains a malicious or misleading instruction and the agent follows it. Fix: treat all tool and retrieval content as untrusted, sanitize before injection, and never allow it to override the system prompt.

Silent tool abandonment. The agent stops calling tools and starts guessing from its training data — correct-sounding, wrong answers, with the confidence of a model that does not know it is guessing. Fix: instrument every run, track tool-call rate, and alert when it drops. A support agent that used tools on 90% of runs and suddenly uses them on 40% is not being more efficient; it is being less honest.

Cost creep. A complex task averages 3–8 tool calls, each with a round-trip and context growth. At a few thousand tasks a day, that is real money. Fix: measure cost per resolved task, cache tool results (a balance lookup rarely changes), and use a cheaper model for the loop with a stronger model only for final synthesis.

Escalation as an afterthought. The loop has no defined way to give up gracefully, so it either loops forever or produces a confident wrong answer. Fix: a written escalation contract — “when uncertain, produce a one-paragraph summary of what was tried and what is missing, and hand to a human.”

When NOT to Build an Agent

I have spent years telling clients not to build what they asked for, and agents are the current champion of that conversation. The decision rule is short:

Build an agent when the task is goal-directed, multi-step, needs tools or data lookups, and changes enough that hand-written rules would be a maintenance nightmare.

Do not build an agent when the task is a single step, the inputs are predictable, or the cost of a wrong autonomous action is higher than the cost of a human clicking a button. I timed a form-filling flow for a client: the deterministic version resolved in 1.4 seconds at about $0.0001 each; the agent version took 6 seconds and about $0.02 each, and occasionally misread a field. The deterministic version won, and the client saved real money by not building what he asked for. That is the honest part of this job: the loop is powerful, and it is not free, and most business processes do not need it.

The Practitioner’s Checklist

Before you call an agent done, run this list:

• [ ] The loop has a step budget and a cost budget, enforced in code, not in a README
• [ ] Every tool has a name, a contract-grade description, and a JSON schema
• [ ] Tool arguments are validated before execution; malformed calls return readable errors
• [ ] Tool results are treated as untrusted data; prompt-injection attempts are sanitized or blocked
• [ ] Working memory is pruned surgically; long-term memory is retrieved, not dumped
• [ ] Retrieval quality has an evaluation set and a measured recall metric
• [ ] Repeated failing tool calls trigger escalation, not another retry
• [ ] A defined escalation contract produces a human-readable summary
• [ ] Every run is logged: steps, tool calls, cost, latency, outcome
• [ ] An alert fires when tool-call rate or retrieval quality drops

The Loop Is the Product

The CTO who asked why his agent kept failing went home with one diagram and one sentence: the loop decides, the runtime disciplines, and the guardrails decide what “done” means. His team spent the next week fixing the scaffolding instead of buying a new framework, and their resolution rate went up for a reason that had nothing to do with the model.

The next time someone shows you an “autonomous agent,” you now know what you are looking at: a loop, some declared tools, memory in two or three layers, and a set of budgets deciding when it may stop. The frameworks will keep coming and going. The loop will not.

*Gulshan Yad