citesure: stop shipping hallucinated citations

2026-07-23 · SybilGambleyyu

In April 2026, Nature reported that tens of thousands of 2025 publications may include invalid references generated by AI. arXiv will ban submitters for a year when a paper shows incontrovertible evidence of unchecked LLM output — including fabricated references. ICLR 2026 audits found roughly one in twenty-nine accepted papers carried at least one likely hallucinated citation.

The hard part is not “use AI less.” Authors already use assistants to fill BibTeX, polish related-work sections, and expand paraphrased cites into full entries. The hard part is a gate that fits real writing workflows: local CLI, CI, structured reports, and repair — not only a one-off web form after the damage is done.

What existing tools leave open

Several good projects check whether a cited work exists (RefChecker, Hallucinator, CheckIfExist, Paperpile’s Citation Checker). Reviewers need those. Authors and automated agents need something slightly different:

Fail a PR or pre-submit hook with a clear exit code
Emit SARIF/JSON for dashboards and code scanning
Rewrite BibTeX fields from authoritative metadata while keeping citation keys
Flag retractions and poisoned metadata (right title, wrong authors/year)
Install in one line and run offline-friendly unit tests without a browser

What citesure does

citesure is an open-source Python library and CLI that verifies bibliographies against Crossref, doi.org, arXiv, and PubMed (optional Semantic Scholar / OpenAlex), then optionally repairs them.

pip install git+https://github.com/SybilGambleyyu/citesure.git

citesure check refs.bib
citesure check refs.bib -f sarif -o citesure.sarif
citesure fix refs.bib -o refs.fixed.bib

Statuses you get per entry:

verified — strong match, consistent metadata
metadata_mismatch — real work found; fields disagree (classic LLM “autocomplete” failure)
retracted — Crossref-style retraction signals
not_found — no acceptable match (candidate hallucination or unindexed work)
ambiguous — several close matches; human review

Default CI fails on not_found, retracted, and errors. Use --fail-on-mismatch or --strict when you want a tighter gate.

Evidence on a mixed bibliography

Against a fixture with two real papers, one deliberate fabrication, and one entry with a real title but wrong authors/year, the CLI verifies the real works, marks the fabrication not_found, and surfaces the poisoned metadata for repair. Live API tests in the repo exercise the same path on every release candidate.

Try it in the browser

Demo (Cloudflare Worker, same idea as the CLI, polite Crossref/arXiv lookups): citesure.sybilgambleyyu.workers.dev

What it is not

citesure flags suspicion, not guilt. Databases miss books, brand-new preprints, and many non-English venues. A not_found result is a lead for a human to check — never an automatic accusation. Claim-level support is available via citesure claim (lexical abstract/title overlap, plus optional SpaceXAI/xAI judge with --llm). That checks citation relevance, not absolute truth. Existence and metadata integrity remain the primary gate.

What shipped since launch

PDF bibliography extraction (citesure check paper.pdf)
Claim-support scoring (lexical + optional LLM judge; mini-bench 29/29)
MCP server for AI coding agents
Overleaf/GitHub Action and pre-commit templates for paper repos
--used-in to check only keys cited in the manuscript (huge shared .bib files)
extract-claims to pull claim sentences near citations for relevance audits
Open Library / ISBN book verification; arXiv disk cache for rate-limit resilience
Offline dedupe, merge, export-csl; journal CI templates
CJK-aware title matching for Chinese/Japanese/Korean bibliographies
CourtListener verification for US case law (reporter cites like 347 U.S. 483)
Labeled evaluation set: 250/250 on citation-integrity-bench (CS/ML, clinical guidelines, law-and-economics, US SCOTUS cases, CJK titles, German physics titles, retractions, fabrications)