STROBE/CONSORT Compliance via LLM Audit: Catching Reporting Gaps Before Reviewers Do
Reporting checklists invite false confidence. A manuscript may mention randomization without explaining sequence generation, or mention missing data without reporting how many observations were missing. A language model can help locate these gaps, provided it is asked to quote evidence rather than award a vague “compliant” score.
It is equally important to use the current document. CONSORT 2025 supersedes CONSORT 2010 for randomized trials. STROBE remains the core reporting guidance for cohort, case-control, and cross-sectional studies, with extensions for specific designs.
Why LLM Audits Work for Checklists
The first pass is retrieval: where, if anywhere, does the manuscript address each item? The second is specificity: does the quoted text contain what the item requests? Separating these tasks makes unsupported inference easier to detect.
A statement such as “participants were randomized” is present, but it does not explain how the allocation sequence was generated or concealed. The model should mark that distinction rather than silently giving credit.
The Audit Prompt Template
Download the applicable checklist from its official site. Provide the checklist and the complete manuscript as clearly labelled sources, including tables, figures, supplements, and protocol references when available.
Pass 1 — evidence map
For every checklist item, return: item number; status (present, partial, missing, or not applicable); an exact quotation from the manuscript; and page, section, table, or figure location. If no text supports the item, mark it missing. Do not infer information.
Pass 2 — challenge the map
Review each “present” item against the checklist wording. Identify any requested element not established by the quotation. Downgrade the item to partial when necessary. Do not assess methodological quality unless the checklist explicitly asks for that information.
Export the result as a table. A human author should verify every quotation and resolve each partial or missing item in the manuscript before completing the journal’s checklist.
STROBE vs CONSORT: Picking the Right Checklist
CONSORT is for randomized trials; the 2025 statement includes a 30-item checklist and flow diagram, with extensions where relevant. STROBE is for the main observational designs. Other designs require other guidance: PRISMA for systematic reviews, STARD for diagnostic-accuracy studies, and CARE for case reports.
Check the target journal’s instructions as well as the official guideline site. The model should not choose the reporting standard without human confirmation.
What the LLM Misses
Three boundaries matter:
- Reporting is not validity. Finding a sample-size paragraph does not establish that the calculation is correct.
- Missing inputs stay missing. An audit without tables, figures, and supplements will misclassify items reported there.
- Consistency is a separate test. Checklist coverage does not reveal every contradiction between abstract, methods, results, registry, and protocol.
Use the LLM for the evidence map. Use a statistician, methodologist, and author review for the judgment layer.
Fix gaps without inventing methods
When an item is missing, the model should draft a question before it drafts prose. “How was allocation concealed?” is safe; inventing a central randomization service is not. The author can answer from the protocol, registry, statistical analysis plan, or study records, after which the model may help place the verified detail in the manuscript.
If the information was never recorded or the procedure was not performed, report that limitation honestly. A reporting audit is meant to expose the study record, not retrofit an idealized method after the fact.
One More Use: The Submission Checklist for R2
Repeat the map after a major revision. Moving or condensing sections can create reporting gaps even when the underlying study has not changed. Compare the revised evidence map with the previous one, then submit the completed checklist with page references that match the final manuscript.
For adjacent checks, see AI-Assisted Systematic Reviews and Self-Peer-Review with AI.
The Checklist: From Idea to Submission provides a reusable sequence of pre-submission gates. Pair it with the official checklist for the actual study design; a workflow aid should never replace the reporting standard itself.