Last year I started building Acurio, a tool that verifies citations in academic theses against their sources. One pattern kept showing up in real student theses: bibliographies full of references that simply don’t exist. Plausible-looking authors, plausible journal, plausible year — and no matching record anywhere. Most of them were LLM hallucinations the student never caught.
The existence check turned out to be the one piece that doesn’t need any AI at all. So I extracted it into a small open-source CLI: citecheck.
What it does
Point it at a .bib, .ris, or CSL-JSON file — the export you get from Zotero, Mendeley, or EndNote — and it checks every reference against three public scholarly APIs:
-
Crossref and OpenAlex — does a matching record exist at all? A wrong DOI, a typo’d title, or a hallucinated reference shows up as
not found. - Crossref retraction metadata — has the work been retracted? (You’d be surprised how often Wakefield 1998 still gets cited straight.)
- DOAJ — is the journal a listed open-access journal? Shown as a positive signal.
$ npx citecheck references.bib
✓ verified watson1953 Molecular Structure of Nucleic Acids…
~ partial wakefield1998 Ileal-lymphoid-nodular hyperplasia… ⚠ RETRACTED
✗ not found fabricated2099 Quantum Entanglement of Bibliographic Phantoms…
Summary: 1 verified · 1 partial · 1 not found · 1 retracted
Enter fullscreen mode Exit fullscreen mode
No API key, no signup, nothing uploaded. It only sends each reference’s DOI/title to the public APIs and prints the result.
Design decisions worth stealing
Auto-detect the input format. Users don’t know (or care) whether their export is BibTeX, RIS, or CSL-JSON. The CLI sniffs the format, including from stdin: cat refs.bib | citecheck -.
DOI first, title second. If a reference has a DOI, that’s a precise lookup. No DOI (books, older works, grey literature) falls back to a title search with fuzzy matching — which is also where the honest uncertainty lives, hence the partial verdict instead of pretending binary confidence.
Be conservative about “not found”. A missing record can mean a hallucinated reference — or a book that Crossref just doesn’t cover. The output distinguishes “no match anywhere” from “partial match”, because a false accusation of fabrication is worse than a missed one.
No build step at install. The npm package ships prebuilt, so npx citecheck cold-starts fast.
Why this matters right now
Every study on LLM-generated citations finds fabrication rates that should terrify anyone grading theses — GPT-4o still fabricates roughly one in five references in some benchmarks. The scary part isn’t that students use LLMs; it’s that a hallucinated reference looks exactly like a real one until someone actually resolves it.
Checking existence is the cheap, deterministic 80%. (Checking whether the source actually supports the claim citing it is the hard part — that’s what the hosted product does.)
Try it
npx citecheck references.bib
Enter fullscreen mode Exit fullscreen mode
Repo: https://github.com/tobiasosDev/citecheck — MIT licensed. Feedback and PRs welcome, especially around false positives on non-DOI sources: books, grey literature, and non-English titles are the current weak spots.
답글 남기기