Building Your First AI Agent from Scratch (No Framework)

How to build a production-shaped agent in ~150 lines of Python — no LangChain, no CrewAI, nothing you cannot read line by line.

A startup founder asked me a question I hear constantly: “Why are we paying for an agent framework when our feature is basically ‘call the model, call an API, retry’?” He was not being cheap. He had watched a demo where a framework “did everything,” then spent two weeks fighting its abstractions when his workflow did not fit the framework’s opinion of how agents should work. The framework was not wrong. It was just a guess about his problem, and his problem was more specific than the guess.

So I did what I always do when a framework is in the way: I built the agent by hand. Around 150 lines of Python, no dependencies beyond an OpenAI-compatible HTTP client. When I opened the file for him, every token was traceable — he could see exactly what went into the context, what the model returned, and when the loop decided to stop. He shipped it to production the following week, and it is still running.

This article is that build, step by step. You will end with an agent that takes a goal, uses tools, has working memory, respects budgets, and escalates when it is out of its depth — and you will understand every line of it. Once you have built one of these by hand, every framework stops being magic and becomes a set of opinions you can evaluate.

Why Build by Hand (When Frameworks Exist)

Let me be honest about the trade-off, because there is one. Frameworks like LangChain and CrewAI compress months of patterns into configuration, and for a standard workflow — chat with retrieval, a few tools, an orchestrator — they can genuinely save you a week. The compressed version is also the version you cannot read: when a tool call misbehaves, the stack trace points into the framework’s internals, and the framework’s memory strategy is a design decision you inherited, not one you made.

Building by hand buys you three things you cannot get from configuration:

1. Readable context. You see every token that enters the model. When the agent behaves oddly, you can reproduce it, because you control the assembly.
2. Honest budgets. Step limits, cost limits, and escalation are your code, not a flag somewhere in a framework’s docs that you may never find.
3. Debuggable failures. The run log is yours. You know exactly what the agent tried, why it tried it, and where it gave up.

The cost is that you write the loop yourself — which is about fifty lines. The rest of this build is tools, memory, and guardrails, which you would write inside a framework anyway.

The Architecture We Are Building

Before code, the shape of the thing. Our agent runs a loop with four components:

        ┌─────────────────────────────────────────────┐
        │                  AGENT LOOP                 │
        │                                             │
 User goal ──▶ assemble context ──▶ model decides ──┐ │
        │                             │             │ │
        │                  answer? ──▶ return       │ │
        │                  tool call ─▶ execute ────┼─┘
        │                             │             │
        └─────────────────────────────┼─────────────┘
                                      ▼
                            budget guards & escalation

Enter fullscreen mode Exit fullscreen mode

• Tools are declared functions the model can invoke, executed by our runtime.
• Memory is retrieved context injected before the model decides.
• Guardrails decide when the loop may continue and when it must stop.
• Escalation is a defined hand-off with a readable summary.

Step 1: The Tool Layer

The first thing to build is the mechanism that turns a model’s structured request into a real function call. I keep it boring: a registry of tools, each with a name, a description, a JSON schema, and a Python callable.

import json
from typing import Callable, Any

class Tool:
    def __init__(self, name: str, description: str, schema: dict, fn: Callable):
        self.name = name
        self.description = description
        self.schema = schema
        self.fn = fn

    def to_openai_spec(self) -> dict:
        return {
            "type": "function",
            "function": {
                "name": self.name,
                "description": self.description,
                "parameters": self.schema,
            },
        }

    def call(self, arguments: str) -> str:
        try:
            args = json.loads(arguments)
            result = self.fn(**args)
            return json.dumps(result)
        except (json.JSONDecodeError, TypeError, KeyError) as exc:
            return json.dumps({"error": f"invalid tool call: {exc}"})


def lookup_order(order_id: str) -> dict:
    # Production: query your orders DB here, with authz and caching.
    return {"order_id": order_id, "status": "paid", "amount": 14900}


TOOLS = [
    Tool(
        name="lookup_order",
        description=(
            "Look up an order by its ID. Returns status, amount in paise, "
            "and delivery status. Raises an error if the order does not exist."
        ),
        schema={
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
        },
        fn=lookup_order,
    ),
]

Enter fullscreen mode Exit fullscreen mode

Two details matter. The description is a contract for the model, not a comment for humans — the model reads it to decide when to call this tool, so it must state preconditions. And the call() method never lets an exception crash the loop; a bad call becomes a readable tool result that the model can recover from.

Step 2: Memory as Retrieved Context

Next, memory. I am going to keep this minimal but production-shaped: an in-memory store of past facts, retrieved by the model’s own sense of relevance via a vector index. For a first agent, you do not need a database server; you need the layer, and you need it to be the thing you swap later.

class SimpleMemory:
    def __init__(self, embed: Callable[[str], list[float]]):
        self.embed = embed
        self.items: list[tuple[str, list[float]]] = []

    def remember(self, text: str) -> None:
        self.items.append((text, self.embed(text)))

    def recall(self, query: str, top_k: int = 3) -> str:
        if not self.items:
            return "(no memory yet)"
        q = self.embed(query)
        scored = sorted(
            self.items,
            key=lambda item: _cosine(item[1], q),
            reverse=True,
        )
        return "\n---\n".join(text for text, _ in scored[:top_k])

Enter fullscreen mode Exit fullscreen mode

For the embedding function, use any OpenAI-compatible endpoint with text-embedding-3-small or a local model. In production this becomes pgvector or Qdrant; the interface — remember() and recall() — is what survives the swap. That is the real value of building the layer yourself: you own the seam.

Step 3: The Loop with Guardrails

Now the heart. The loop assembles context, calls the model, and interprets the response. The guardrails are not an afterthought here; they are written into the loop itself so they cannot be skipped.

from openai import OpenAI

class Agent:
    def __init__(self, model: str, tools: list[Tool], memory: SimpleMemory,
                 client: OpenAI):
        self.model = model
        self.tools = {t.name: t for t in tools}
        self.memory = memory
        self.client = client

    def run(self, goal: str, max_steps: int = 6,
            max_cost_cents: float = 10.0) -> dict:
        system = (
            "You are an assistant that completes a goal using available "
            "tools. Rules: only call a tool when you need data; never invent "
            "tool results; if you cannot finish, escalate with a summary of "
            "what you tried and what is missing."
        )
        history = self.memory.recall(goal)
        messages = [
            {"role": "system", "content": system},
            {"role": "user", "content": f"RELEVANT PAST CONTEXT:\n{history}\n\nGOAL: {goal}"},
        ]
        steps = 0
        cost = 0.0
        for steps in range(1, max_steps + 1):
            response = self.client.chat.completions.create(
                model=self.model,
                messages=messages,
                tools=[t.to_openai_spec() for t in self.tools.values()],
            )
            cost += _estimate_cost(response)  # tokens * rate
            if cost > max_cost_cents:
                return {"outcome": "escalated",
                        "summary": "cost budget exceeded",
                        "steps": steps, "cost_cents": cost}

            msg = response.choices[0].message
            if not msg.tool_calls:
                self.memory.remember(f"goal: {goal} -> answer: {msg.content}")
                return {"outcome": "done", "answer": msg.content,
                        "steps": steps, "cost_cents": cost}

            messages.append(msg)
            for call in msg.tool_calls:
                tool = self.tools.get(call.function.name)
                result = (
                    tool.call(call.function.arguments)
                    if tool else json.dumps({"error": "unknown tool"})
                )
                messages.append({
                    "role": "tool",
                    "tool_call_id": call.id,
                    "content": result,
                })

        return {"outcome": "escalated",
                "summary": "step budget exhausted, tried: " +
                           repr([m.get("content", "")[:80] for m in messages[-4:]]),
                "steps": steps, "cost_cents": cost}

Enter fullscreen mode Exit fullscreen mode

Read the guardrails, because they are the product: a step budget so the loop cannot run forever, a cost budget that fails soft, unknown-tool handling so a hallucinated tool name cannot crash the run, and a memory write-back so the next goal starts from what this run learned. Escalation returns a readable summary — that is what the human receives, not an error trace.

Step 4: Wiring It Together

def embed(text: str) -> list[float]:
    r = client.embeddings.create(model="text-embedding-3-small", input=text)
    return r.data[0].embedding

client = OpenAI()  # any OpenAI-compatible endpoint

memory = SimpleMemory(embed=embed)
agent = Agent(model="your-model", tools=TOOLS, memory=memory, client=client)

result = agent.run("Where is order ORD-9911 and has it been paid?")
print(result)

Enter fullscreen mode Exit fullscreen mode

Run that and the agent will call lookup_order, get the status, and answer — or escalate if it cannot. That is the entire skeleton. It is small because the loop is small. Everything you will ever add — a vector database, retries, multiple agents, a permission layer — attaches to one of the seams you just built.

Production Reality: What I Broke on My First Build

I have built enough of these to know the exact ways the naive version fails, and you will hit them too. Here is the list in the order they will find you:

1. The tool description was too lazy. My first lookup_order said “gets order info,” and the model called it for users’ own made-up order IDs. Rewriting the description to state preconditions and error behavior fixed most misuse without any code change. Treat descriptions as documentation, because that is what the model reads.

2. Malformed JSON killed the run. The model occasionally emitted truncated JSON arguments, and my first version threw, killing the loop. The Tool.call() error path — returning a readable error instead of raising — is what saved it. A model that gets a tool error can recover; a loop that crashes cannot.

3. The cost guard was the first thing I removed, and I regretted it. In testing, a single pathological run hit hundreds of tool calls before I noticed. Put the cost budget in from the first line, not after the invoice.

4. Memory write-back polluted later runs. I was remembering every raw goal, and after a few sessions the recalled context was dominated by noise. Fix: remember distilled facts (“order ORD-9911: paid, delivered”), not raw user messages, and cap the tokens you inject.

5. No evaluation for retrieval. When I swapped the embedding model, recall quality drifted and I did not notice for a week. Keep a fixed set of test goals with the facts you expect retrieved, and run it on every change.

Scaling Up: Second Tool, Retries, and a Permission Layer

Once the skeleton works, the natural next questions are about scale. Here is the honest path from 150 lines to something a client will pay for, and where each addition attaches to a seam you already built.

Adding a second tool is a registration, not a redesign. Define the function, write a contract-grade description, add it to the TOOLS list, and the loop does the rest. The model will start choosing between tools based on descriptions, which is exactly why descriptions are the highest-leverage documentation you will ever write. When I added send_refund to an agent that already had lookup_order, the model learned to call lookup_order first to confirm eligibility, then send_refund — two steps it had never been shown, driven entirely by the descriptions.

Retries need to be explicit, or the loop invents them. A naive loop that hits a rate-limited tool will call it again on the next step, burning budget and latency. Add retry behavior in one of two places: at the tool layer, where a transient failure (429, 503) returns a specific error the model can react to; or in the loop, with a fixed attempt cap. Do not let the loop “decide” to retry by rephrasing a call — that is the greedy-loop failure that spends real money discovering nothing.

A permission layer changes the loop, not the tools. For mutating actions — send an email, refund money, delete a record — the agent should not execute directly. The pattern: mark the tool as requires_approval: true in its metadata, and when the loop reaches it, return a pending result and hand the human a summary with a yes/no gate. The tool function stays the same; the runtime decides whether the call executes or waits. I have shipped agents where every write action flows through this gate, and it is the difference between “the agent acted” and “the agent proposed, a human approved.”

Multi-agent is last, not first. Resist the urge to turn your working single agent into a team. Every agent boundary is a handoff, and every handoff loses context and burns tokens. Split only when a real constraint demands it — a read-only researcher with different permissions than a write-capable operator, for example — never for aesthetics. The single loop you can trace will beat the five-agent crew you cannot debug.

When NOT to Build by Hand

The honest counterpart: do not hand-roll an agent when your problem is a standard pipeline and a framework’s opinion matches it exactly — retrieval chat, a fixed tool surface, a single orchestrator. You will spend the same time writing glue either way, and a framework’s ecosystem (memory integrations, tracing, deployment) saves real effort. Hand-roll when the workflow is unusual, when you need to trace every token, or when a framework’s abstractions are costing you more than they save.

The decision rule: if you can describe your loop in a paragraph, write it by hand. If you cannot, you are not ready to delegate it to a framework either.

The Checklist Before You Ship

• [ ] Tool descriptions state preconditions and error behavior
• [ ] Malformed tool arguments return readable errors, never crash the loop
• [ ] Unknown tool names are handled gracefully (escalate, don’t crash)
• [ ] Step budget and cost budget are enforced inside the loop
• [ ] Escalation returns a human-readable summary of what was tried
• [ ] Memory has a recall seam you can swap for a vector DB later
• [ ] Memory writes distilled facts, not raw logs
• [ ] Tool results are treated as untrusted input to the model
• [ ] An eval set measures retrieval quality on every model/tool change
• [ ] The full run is logged: steps, tool calls, cost, latency, outcome

The 150 Lines Changed His Mind

The founder who asked why we were paying for a framework got his answer the moment the file opened: the loop was six lines, the tools were a registry, the memory was a seam, and the guardrails were explicit. Two weeks later the same argument saved another client a migration they did not need. The loop is not the hard part — it never was. The hard part is deciding what goes into the context, when the loop may stop, and what a graceful failure looks like. Those are decisions, and they belong in code you can read.

Write your first agent by hand. Read every token. Break it on purpose. When you understand the loop well enough to trust it, you will know exactly when a framework is helping you — and when it is just guessing.

*Gulshan Yad