Citation Verification with CiteCheck: A 5-Minute Setup
CiteCheck started as a script I needed for ARM1 — I had a 45-reference manuscript drafted with Claude and no fast way to verify whether each citation was real and whether the claim I was making actually matched what the paper said. A few hours and a lot of CrossRef API calls later, I had something that worked. I posted it to PyPI because every researcher I showed it to wanted it. MIT license, no signup, no API key.
The Two Citation Problems It Solves
Most researchers think about fabricated references — DOIs that don't exist, author names that don't resolve anywhere. CiteCheck catches those. But that's the obvious failure mode, and it's not the one that worries me most.
The harder problem is alignment. A paper exists, the DOI resolves, but the claim in your manuscript doesn't match what the paper found. This happens when an LLM drafts prose from a paper's abstract tone rather than its actual results section. Three fabricated DOIs in a colleague's Claude-drafted manuscript looked completely real — standard author names, plausible journal titles, formatted correctly. All three would have survived human peer review. CiteCheck caught them in under two minutes.
Install and Run
pip install citecheck
Python 3.10+ required. Nothing else.
To check a manuscript:
citecheck --input manuscript.txt --report json
CiteCheck queries CrossRef, PubMed, Semantic Scholar, and OpenAlex for each reference — combined index of approximately 240M papers. For each citation, the report returns: existence status, metadata match (author list, year, title), and a claim-alignment flag where an LLM-judge layer compares your cited claim against the paper's abstract.
The alignment flag is probabilistic, not binary. It signals that something warrants a manual check — not that the citation is wrong. Every flagged item takes under a minute to verify by pulling the abstract. That's a better use of 5 minutes than a desk rejection for a fabricated reference.
GitHub Action Integration
If you version-control your manuscripts (and for systematic reviews, this should be standard practice), the GitHub Action runs CiteCheck automatically on every commit:
- uses: tuyentran-md/citecheck-action@v1
with:
input: manuscript.txt
This catches references that break mid-draft — a problem more common on collaborative manuscripts where sections get repasted from different source documents. One author edits a bibliography entry in their local copy; the pasted version in the shared draft becomes stale. The Action flags it before it reaches a submission portal.
What It Doesn't Do
CiteCheck covers the citation-fidelity layer only. It doesn't evaluate your argument structure, flag statistical errors, or check CONSORT/STROBE compliance. Use it as one gate in a pre-submission workflow, not as the entire workflow.
It also doesn't cache between runs by default — each manuscript check triggers one API call per reference across the backends. On a standard connection, a 60-reference manuscript takes under two minutes. On hospital network infrastructure with packet inspection, it can be slower; the Python API mode with a local cache directory helps there.
For the bigger picture on where citation verification fits in the clinical research AI stack, see The 2026 AI Tool Landscape for Clinical Researchers. For the mechanics of why LLMs fabricate citations in the first place, Citation Hallucination: The Silent Killer in AI-Assisted Writing covers that in detail.
The Honest 5-Minute Caveat
The install takes 5 minutes. The first run on a real manuscript takes closer to 20: install, run, read the report, understand what each flag means, check the flagged items. Still worth it. A retraction is worse than 20 minutes.
pip install citecheck — MIT-licensed, 240M-paper index, runs locally.