TIL – Choosing Between Code, an LLM Call, and an AI Agent

Two questions sent me down this path.

Question one: what is an “AI agent,” really? Most job posts mention them. I had not looked into it deeply, and from the outside I could not tell what it referred to. Is an agent a different endpoint? A different model? A library we install? Or is it just a name for calling an LLM API in some particular way?

Question two came later, and it stopped me cold. Once I understood the mechanics and sat down to pick something to build, I could not come up with a single idea that I could not also write as ordinary Go: a function that calls some APIs, hits a database, and returns a result. If plain code can do it, why pay a model to do it worse?

This is what I found. There is a real answer to both, and the second one has a trap in it.

So I built both

Code: github.com/Arkoes07/llm

One Go service, one interface, four endpoints of increasing complexity:

POST /chat/no-memory        one stateless LLM call
POST /chat                  multi-turn, history per session
POST /chat/agent/weather    tool-calling loop, one tool
POST /chat/agent/log-triage tool-calling loop, three tools

Enter fullscreen mode Exit fullscreen mode

No framework. Hand-rolled net/http first, then a second implementation behind the same interface using an SDK, so I could see exactly what the SDK was doing for me. (Answer: it removed boilerplate, not thinking. The loop logic was identical in both). Worth noting: Groq’s official SDKs are Python and JS only, so in Go we are on community libraries or an OpenAI-compatible client.

Starting at a plain call and walking up to an agent one step at a time turned out to be the whole answer to question one.

An agent is a while-loop

Here is the version with no vocabulary.

A plain LLM call is a stateless HTTP request. We send messages, we get text back. The model remembers nothing. “Conversation history” is state we store and resend every time. That is /chat/no-memory and /chat above, and the only difference between them is who keeps the array.

An agent adds one thing: we also send a list of tools, described as JSON schemas. Now the model can respond with “call get_metrics with {"service":"checkout"}” instead of a final answer. So we loop:

1. Send messages plus tool schemas.
2. Model replies with either a final answer, or a tool request.
3. Our code executes the tool. The model executes nothing, it only asks.
4. Append the result to the history, call again.
5. Repeat until it stops asking, or we hit an iteration cap.

That is the difference. Same endpoint, same model, about 150 lines of extra Go. Every agent framework is this loop plus state persistence, retries, and tracing.

What I realized while building it is that most of the hard parts are things we already know under different names:

Agent problem What it actually is Tool dispatch + schema validation RPC dispatch Model calls the same tool twice Idempotency Multi-step work that must not half-complete Saga / compensation Context window overflow Bounded cache with an eviction policy Untrusted text in the prompt issuing instructions An authz boundary Runaway loop Circuit breaker “Did this actually work?” Regression testing, for non-deterministic output

There is a reason the table lines up like that, and the loop hides it.

From inside the code it looks like one API call repeated. But we call the model, the model asks for a tool, we call that tool (in a real system, another service: a logging backend, a metrics API, a runbook store), then we call the model again. One request fans out across several independent services, in an order nobody fixed in advance. That is orchestration, and we already have a name for it: a distributed system.

The difference is which part we cannot trust. Normally the orchestrator is our own code, the most reliable thing in the system, and we save our suspicion for the dependencies. In an agent, the orchestrator is the model. The component deciding what to call next is now the least reliable one: it picks a different sequence on identical input, it can emit a malformed request, and when it fails it usually fails plausibly instead of loudly.

So the instincts still apply, we just point them somewhere new. Validate every argument. Make writes idempotent, because the caller may repeat them. Bound the loop. Treat anything a tool returns as data, never as instructions. What we already do at the edge of a system, applied to the orchestration we are used to trusting.

What is actually new is a short list: prompting, structured output, evals, and retrieval.

Question one, answered. Which is when question two showed up.

Why would I ever use this?

I needed something to build. And every idea I had, I could immediately picture as normal Go.

“Generate a study plan” is one LLM call with a structured output, wrapped in a function. No loop. “Classify support tickets” is one call, then a switch on the result. “Summarize a document” is one call. Every time I reached for the loop, plain code was cheaper, faster, deterministic, and testable.

That is not a small objection. It is the whole question. The loop costs us money (tokens), latency (seconds per iteration), determinism (same input, different output), and debuggability (behavior lives in prose, not code). What are we buying with all that?

Exactly one thing: the model chooses the next step at runtime, using information that did not exist when we wrote the code.

Not intelligence, and not language. Those come from a single call. The loop buys runtime branch selection and nothing else.

So the question becomes: when do we actually need that?

The criterion I landed on

The common framing is “simple problems go to code, complex problems go to an agent.” Too vague to act on. Here is the version I would use:

Can we draw the flowchart before we see the input?

If yes, the decision points are enumerable and we can write the branches, so we should write them. Code is deterministic, testable, debuggable, and roughly free. An agent there is pure overhead, and the LLM’s job, if it has one, is a single call inside one box: classify, extract, summarize. No loop.

If no, meaning the path depends on what each step reveals and the real tree has thousands of branches we would only discover in the moment, that is the narrow case where the loop earns its cost.

That criterion is why I picked log triage. Consider:

if latencyHigh {
    checkDatabase()
    if dbSlow {
        checkConnectionPool()
    } else {
        checkDownstream()  // which one? there are 30
    }
}

Enter fullscreen mode Exit fullscreen mode

We can write this. But which downstream service? Depends on what the logs said. What if the DB check reveals a deploy changed a query plan? We did not have a branch for that. The order we investigate depends on what each step returns.

That is the trigger. Not “it is complicated,” but “I cannot enumerate the branches at code-writing time.”

Keep this criterion in mind. The rest of the post is about what happens when we pass it and then slowly undo our own answer.

The test case

A fake incident with a deliberate lie in it:

checkout p99 spiked at 14:03. But checkout is fine. Its DB pool is healthy (4/20) and traffic is normal. It calls payment, whose max_pool_size was quietly lowered 20 → 5 by a config reload at 13:58. Payment saturated its pool and started timing out. Checkout is the victim, not the cause.

Three tools over hardcoded data: query_logs, get_metrics, search_runbook. Getting it right means ruling out checkout’s own resources, hopping to the downstream service, and (the trap) widening the time window past the incident to catch that 13:58 config change.

Mocked data is a feature, not a shortcut. Every failure is then unambiguously the loop’s, not flaky infrastructure.

Then it went wrong, five times

1. It cited a runbook it never opened

iteration 1: query_logs(checkout, 13:55-14:05)
iteration 2: get_metrics(checkout, latency)
iteration 3: get_metrics(checkout, error_rate)
iteration 4: get_metrics(payment, latency)
iteration 5: get_metrics(checkout, request_rate)
iteration 6: query_logs(payment, 13:55-14:05)
iteration 7: get_metrics(payment, db_pool)
iteration 8: get_metrics(checkout, db_pool)

Enter fullscreen mode Exit fullscreen mode

Note what is missing: search_runbook, never called. Now the answer it produced:

…These symptoms match the known runbook pattern for “Database connection-pool exhaustion”.

Remediation (per the runbook)

1. Restore / increase the DB pool size for the payment service
2. Restart the payment service to clear lingering connection-wait states

The root cause was correct. The provenance was invented. It attributed general knowledge to an internal document it never opened, and step 2 was not in my runbook at all.

Nothing in the output looked wrong. Plausible content, authoritative formatting, correct conclusion. The only way to catch it was diffing the answer against the trace. Now imagine a real runbook saying “do NOT restart, drain connections first, page the DBA.”

Fix (system prompt): “Only cite the runbook if you called search_runbook. Otherwise say the recommendation is based on general knowledge.”

The lesson: an agent can reach a correct answer through invalid reasoning and give us no signal that it did. That is the argument for evals, and it is checkable. “Every source cited must appear in the tool trace” is an assertion we can write.

2. It stopped at the first plausible explanation

iteration 1: query_logs(checkout, 13:55-14:05)
iteration 2: get_metrics(checkout, error_rate)
iteration 3: search_runbook("payment.authorize timeout")

Enter fullscreen mode Exit fullscreen mode

Three calls, then it concluded: tune the client timeout, add a circuit breaker. Plausible, and wrong. It never checked whether checkout’s own DB was healthy or whether traffic was abnormal. It gathered evidence for its first hypothesis instead of testing alternatives. A human had to type “continue, check the downstream.”

Fix (system prompt): “Do not stop at the first plausible explanation. Before concluding, rule out alternatives: check whether the service’s own resources are healthy and whether traffic is abnormal.”

3. It searched the runbook before it knew what to search for

iteration 6: search_runbook("payment.Authorize timeout")   ← searched here
iteration 7: get_metrics(payment, latency)
iteration 8: query_logs(payment, 13:55-14:05)              ← found the cause here

Enter fullscreen mode Exit fullscreen mode

It searched using the symptom it had at iteration 6, found the actual root cause at iteration 8, and never searched again. A human does this reflexively: it is pool exhaustion, so pull up that runbook. The loop does not re-plan by default.

Fix (system prompt): “After identifying a root cause, search the runbook again using the root-cause terms, not the original symptom.”

4. The caller emitted an invalid call

{"error":{"code":"tool_use_failed","failed_generation":
"<function=query_logs{\"service\": \"checkout\"}</function>"}}

Enter fullscreen mode Exit fullscreen mode

Right tool, right arguments, malformed syntax. It is missing one > after the function name. A gRPC client cannot do this; the wire format is generated and guaranteed. Here the caller’s ability to form a valid request is itself probabilistic.

It also broke my retry logic:

attempt 1: <function=query_logs{...
attempt 2: <function=query_logs{...   ← identical
attempt 3: <function=query_logs={...  ← one char different
attempt 4: <function=query_logs{...   ← identical again

Enter fullscreen mode Exit fullscreen mode

Four attempts, 15 seconds of backoff, the same malformed output. Retry only helps if the output would differ. At low temperature the model is near-deterministic, so an identical request produces an identical failure.

Fix (code, then model): drop tool_use_failed from the retryable set, so we fail fast instead of burning 15 seconds. Then switch models. I did not see it again on openai/gpt-oss-120b.

5. Context growth is the cost model

Rate limit reached ... tokens per minute (TPM): Limit 8000, Used 6886, Requested 1311

Enter fullscreen mode Exit fullscreen mode

Every iteration resends the entire history, so iteration 8 carries all seven prior tool results. Eight iterations, under a minute, into a hard wall. Not because any single call was large, but because they compound. Log prompt_tokens per iteration; that curve is the most honest thing we can show about agent economics.

Fix (code): exponential backoff on 429, honoring the Retry-After header rather than our own curve. This is a mitigation, not a fix. The growth is inherent to the loop.

The two kinds of guardrail

Look back at the labels on those fixes. Two were code, and code fixes are just engineering: better retry classification, better backoff. Nothing interesting there.

The other three were system prompt. And those three are not the same species.

Type A: guardrails that improve judgment.
Better tool descriptions. “Rule out alternatives before concluding.” “Cite the runbook only if you called it.” These do not shrink the option space, they help the model navigate the same space better. Nearly free. Keep them, add more.

Type B: guardrails that encode the flowchart.
“Always call query_logs before get_metrics.” “After identifying a root cause, search the runbook again.” These remove options. Each one is a branch, written in English instead of Go.

The test is mechanical:

If we can state the guardrail as a deterministic rule, we can write it as code. And if we can write it as code, it does not belong in the prompt.

Fix #3 is mine, and it is Type B. “After identifying a root cause, search again with the root-cause terms” is a sequencing rule. I could enforce it in code: if the loop is about to terminate and search_runbook was never called with the root-cause terms, force one more iteration. Deterministic, testable, free. Instead I wrote a branch in English and paid a model to interpret it.

That is the paradox. Remember what the loop buys us: the model choosing at runtime. Every Type B guardrail takes some of that back. We are paying full price for a capability we are actively suppressing. Push far enough and we have a complete flowchart written in English, executed by a probabilistic interpreter, at a hundred times the cost of an if, with no type checking, no test coverage, and no way to diff a behavior change in code review.

We did not build an agent. We built the world’s most expensive switch statement and moved it out of version control.

And notice the shape of it. I passed the flowchart criterion honestly (log triage really is not enumerable) and then spent three fixes quietly re-enumerating it anyway.

Which means the two ideas in this post were never separate. The criterion tells us whether the flowchart exists before we start. The guardrails tell us whether we were right, after. Same spine, checked from both ends. If we find ourselves writing the flowchart in English one rule at a time, the criterion already had the answer.

The diagnostic

The number of Type B guardrails we need is a diagnostic on whether we picked the right tool.

• The agent works with almost none, which means the problem really was not enumerable. The loop earned its cost.
• We keep adding them and reliability keeps improving. That is not tuning. That is the problem telling us it was enumerable, and we should port those rules back into code and shrink the agent to whatever judgment is left.

One caveat: how many we need is not a fixed property of the problem. Compare the two traces above. Finding #2, the three-call run that stopped at checkout, was llama-3.3-70b-versatile. Finding #1, the eight-call run that ruled out checkout’s pool and traffic, hopped to payment, and widened the window to 13:55 on its own, was openai/gpt-oss-120b. Same tools, same fake world, same prompt. The only change was the model string.

I only ran each model once, so treat this as a signal and not a benchmark. But the swing matters: the same code was a supervised tool on one model and an autonomous one on another. A guardrail we need this quarter may be dead weight next quarter.

(Related: the Llama model I started on was deprecated on Groq partway through this project. Model availability is a dependency that disappears).

The answer to both questions

Three tools, not two. Most of the confusion in this space comes from collapsing the middle one.

1. Pure code, no LLM at all.
The flowchart is drawable and every step is mechanical. Parse, transform, query, branch. If we can specify it, we should specify it. This is still most software.

2. A single LLM call inside code we control.
The flowchart is drawable, but one box in it needs judgment over unstructured input: classify this ticket, extract these fields, summarize this text, draft this reply. One call, structured output, and our code decides everything else. This is the overwhelming majority of real “AI features,” and it is not a lesser agent. It is the correct architecture for an enumerable problem.

3. The agent loop.
We genuinely cannot draw the flowchart, because step three depends on what step two returned and there are too many possibilities to enumerate. Incident triage, code review, open-ended research. Here the loop reaches things branching would not, and we pay for it in tokens, latency, and non-determinism.

The line between 2 and 3 is the flowchart criterion. The line between 2 and 1 is just whether any single step needs judgment over unstructured input.

And the sweet spot inside option 3, which is what the guardrail paradox is really about:

Constrain in code everything that can be enumerated. Leave the loop wrapping only the irreducible judgment.

The agent should be as small as possible. That is the principle I would take into the next one, and it follows from the paradox rather than from measurement.

The observation underneath it is smaller and more concrete. Every idea I started with turned out to be a code problem in disguise. That is what question two was. I could not find a use case that plain Go would not handle until I went looking for one deliberately, and even then the loop only survived because I stopped short of writing the last three rules in code.

What I have not resolved

Two things I could not settle from this build.

Where the crossover actually sits. I can say the loop earns its cost when the flowchart is not drawable, but I cannot yet say what that costs in dollars and milliseconds compared to the deterministic version of the same task. I have the argument and not the numbers.

When to stop adding Type B guardrails and rewrite in code. The diagnostic says a rising count means we picked wrong, but there is no threshold. Two Type B rules is clearly fine. Fifteen is clearly a flowchart. I do not know where the line is, and I suspect it depends on how much the remaining judgment step is worth.

Answering either one properly would mean building the same task twice, deterministic and agentic, and measuring cost, latency, correctness, and the inputs the deterministic version cannot handle at all. I have not done that, so for now both stay open.

If you have hit different failure modes running agents in production, I would like to hear them.