Local-first RAG for privileged legal documents: why citations need verification

Lawyers want the same thing everyone else wants from AI: ask a question, get a useful answer, move faster.

The difference is that legal documents often come with privilege, confidentiality, protective orders, client expectations, and professional responsibility duties. Uploading a contract set, deposition transcript, pleading file, or discovery bundle into a generic cloud chatbot is not a neutral implementation detail. A vendor sits in the data path.

That is the design problem behind DocuChat: an open-source, local-first document chat app for attorneys.

The core idea is simple:

Put the AI next to the documents, not the documents next to the AI vendor.

The architecture

DocuChat is a self-hosted legal document RAG stack:

• Local models through Ollama
• FastAPI backend bound to loopback
• LanceDB embedded vector store
• PyMuPDF / Docling / Tesseract for PDFs, tables, and OCR
• Matter-scoped retrieval so one client/matter cannot bleed into another
• Verified page/span citations before answers are displayed

The query path does not require OpenAI, Anthropic, Google, or any cloud AI API key. After local models are downloaded, the workflow can run offline.

Why ordinary citations are not enough

A citation-looking string is easy for an LLM to generate. That does not mean the cited text actually supports the claim.

DocuChat treats citations as a mechanical verification problem. The model can propose an answer and source quote, but the app checks whether that quote actually appears in the retrieved source text at the claimed page/span. If a claim cannot be verified against the source, it is dropped instead of shown.

That creates a more useful failure mode for legal work:

• Better: “I could not find that in the documents.”
• Worse: a confident answer with a fake or sloppy citation.

What this is not

DocuChat is not an AI lawyer. It does not give legal advice, draw legal conclusions, or act outside the user’s machine.

It is a cited-retrieval assistant for document-heavy work: contracts, transcripts, scanned PDFs, exhibits, pleadings, and matter files.

Why open source matters here

For sensitive workflows, “trust us” is not enough. Legal technologists should be able to inspect the data path, citation logic, storage layer, and model boundaries.

The repo is here:

https://github.com/janderswag/docuchat.app

If you care about local-first AI for private legal documents, a GitHub star/watch helps validate that this is worth building further.