Local LLMs for an Infra-Monitoring Agent: The Ollama think Bug, and Why I Still Chose a Cloud Model

OpenClaw went from a weekend project to one of the most-starred repos on GitHub in under five months, and now everyone’s using it to run their inbox, their calendar, their whole digital life. I wanted the opposite: the smallest possible slice of that ecosystem, running local-first, doing one boring job well: infrastructure monitoring. This is what happened when I actually tried to build that, including an undocumented Ollama bug that ate an evening.

In this article

• The smaller slice I actually wanted
• The stack, bare minimum on purpose
• The Ollama think bug, and the actual fix
• Hallucination, not capability, was the real blocker
• Why this matters beyond one setup
• What’s next

The smaller slice I actually wanted

OpenClaw is the reason “AI agent” stopped meaning a chatbot and started meaning something that reads your email, files your GitHub issues, reschedules your calendar, and runs semi-autonomously through a Discord or Telegram interface. It’s impressive: full agent fleets, OAuth into a dozen services, voice mode, phone apps. It’s also the wrong shape for something small enough to trust running unattended near infrastructure I’m responsible for.

That’s what pulled me toward PicoClaw, a much leaner, CLI-first agent runtime that keeps the same core idea (a model, a toolset, a sandboxed workspace, a channel to talk to it through) without the sprawl. No inbox integration, no calendar, no dozen-service OAuth surface. Just an agent loop pointed at a workspace folder and a set of tools I explicitly allow.

The install was the easy part. The real work started with deciding which model to actually trust running inside it.

The stack, bare minimum on purpose

Agent runtime:  PicoClaw (CLI, no web UI, no Docker)
Tier 1 local:   tested — not adopted yet (see below)
Tier 2 local:   Qwen3, no-think config
Tier 3 cloud:   DeepSeek V4 Flash, via a self-hosted LiteLLM proxy

Enter fullscreen mode Exit fullscreen mode

“No Docker” up there isn’t a stylistic choice. I tried the Docker install first, since that’s what most of the setup guides default to. On Mac, PicoClaw’s Docker path expects the model endpoint at host.docker.internal, and getting that to line up with a local Ollama instance and a tunneled cloud endpoint at the same time turned into more plumbing than it was worth. The native binary just uses localhost for everything. Docker is abandoned for this setup entirely now just for simplicity.

PicoLM (a tiny, instant-response local model meant for trivial one-word answers) is sitting installed and untouched. I don’t have a use case for it yet in an infra-monitoring context; “what time is it” isn’t the problem I’m solving. It’s on the list to revisit once there’s an actual low-stakes, high-frequency task worth routing to something that fast.

The real work went into everything above tier 1.

The Ollama think bug, and the actual fix

I went in expecting a local model (something like Gemma4 or Qwen3, running entirely on-device) to be good enough for real analysis work. A lot of people online are running exactly that combination successfully for agent tool-use, and I don’t doubt them. I tried several variants of Gemma4 and a few Qwen builds, specifically stress-testing tool-calling reliability rather than just chat quality, since an agent that can’t reliably invoke read_file or run_command is useless no matter how articulate its prose is.

Reasoning mode has to be turned off, and not the way you’d expect. Qwen3’s default “thinking” output is great for open-ended reasoning and actively harmful for an agent loop that expects a clean, immediate tool call: the model talks itself in circles before ever calling anything. The obvious fix is a runtime flag:

PARAMETER think false

Enter fullscreen mode Exit fullscreen mode

That throws Error: unknown parameter 'think'. Ollama’s Modelfile syntax doesn’t support it at all, despite it looking exactly like every other PARAMETER line that does work. It’s not documented anywhere obvious, and it’s an easy hour to lose assuming you’ve got a typo.

The actual fix has to happen in the prompt template, not the parameters block. You build a custom model from a Modelfile whose TEMPLATE does three specific things:

1. Appends /no_think to every single user message before it reaches the model
2. Strips any <think>...</think> block out of how assistant responses get rendered
3. Forces an empty <think>\n\n</think> pair at the start of every assistant turn, which signals to Qwen3 that the “thinking phase” is already done, so it goes straight to content or a tool call

cat > /tmp/qwen3-nothinker.modelfile << 'EOF'
FROM qwen3:8b

TEMPLATE """
{{- if or .System .Tools }}<|im_start|>system
{{ .System }}
{{- end }}
{{- range .Messages }}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }} /no_think<|im_end|>
{{- else if eq .Role "assistant" }}<|im_start|>assistant
{{ .Content }}<|im_end|>
{{- end }}
{{- end }}<|im_start|>assistant
<think>

</think>

"""

PARAMETER repeat_penalty 1
PARAMETER temperature 0.6
PARAMETER top_k 20
PARAMETER top_p 0.95
EOF

ollama create qwen3-local -f /tmp/qwen3-nothinker.modelfile

Enter fullscreen mode Exit fullscreen mode

(trimmed for readability — the full template also handles tool-call formatting and multi-turn history)

Then point the agent config at qwen3-local, not the vanilla qwen3:8b tag. The custom build is a separate named model in Ollama, so nothing about the fix is implicit. A quick ollama run qwen3-local "hello" confirms there’s no stray <think> block leaking into the response before wiring it into the agent.

Hallucination, not capability, was the real blocker

Once tool-calling was reliable, the real problem showed up: trusting the answer. On multi-step or multi-file analysis, local models would confidently report things that weren’t there: files that didn’t exist, log lines that didn’t match, conclusions that sounded plausible and were simply wrong. For a general assistant that’s an annoyance. For something judging whether infrastructure state looks normal, a confidently wrong answer is worse than no answer.

That’s the actual reason DeepSeek V4 Flash via a cloud API ended up as the tier-3 model. Not because local models are bad (plenty of people are getting Gemma4 and Qwen to work well for exactly this kind of agent), but because reliability and low hassle mattered more to me than keeping every call on-device. It’s a pragmatic call, not a verdict on local models generally, and testing Gemma4 and Qwen further for lower-stakes tasks is still on the list.

Why this matters beyond one setup

The bigger pattern here is one a lot of people building on the OpenClaw wave are going to hit eventually: the agent framework is rarely the hard part anymore. Wiring up a CLI agent, a sandbox, and a model is a weekend. The actual engineering is in the boring middle layer: deciding what a model is trustworthy enough to be handed, tier by tier, task by task, and being honest when a shinier local-only setup isn’t actually the more reliable one.

For infrastructure monitoring specifically, that boring middle layer matters more than usual. Wrong output in a chat app is a bad reply. Wrong output feeding an automated check against production infrastructure is a false sense of security, arguably worse than not automating it at all.

What’s next

The model-routing decisions above were the prep work, and they weren’t the only surprise. The web-channel side of PicoClaw behaves differently from the CLI that tripped me, even different ways of picoclaw install on Macs which is worth its own writeup rather than a footnote here.

The real test comes in Part 2: pointing this agent at a live domain health audit, cloud-brain doing the analysis. It didn’t just confirm the setup worked. It surfaced real, ongoing configuration drift, the kind that accumulates in any environment over time: privileged group membership that had grown stale, a service running under an account it had no business running under, Group Policy scoped to the wrong group entirely. None of that was the point of the exercise, and all of it turned out to matter more than the exercise itself.

Part 2 covers what the audit caught, what the agent got wrong, and whether the DeepSeek-for-reliability bet held up outside of clean test conditions.

Part 1 of a short series on building a minimal, local-first AI agent stack for infrastructure monitoring. Stack: PicoClaw (CLI-only), Qwen3 (local tier, no-think config), DeepSeek V4 Flash via a self-hosted LiteLLM proxy (cloud tier).