Python + Pandas + Claude for Clinical Data Wrangling: The Reproducible Pipeline

Python clinical data analysis with AI assistance handles the boilerplate well: reshaping wide to long, parsing dates, joining encounter tables. It misses the parts that are specific to a hospital extract — the coding scheme that changed between admission years, the lab value stored as text because one site wrote a less-than sign, the patients missing an outcome because they transferred rather than because someone skipped a field. A model reads the column names and the request. It does not know your registry.

Where generated cleaning code goes wrong

Dates that are not one format

An export pulled from more than one hospital system carries more than one date convention, and day-first versus month-first is ambiguous for the first twelve of every month. Ask for date parsing and the model picks one parser and moves on. The ambiguous rows become a silent NaT, and nothing in the output points back to them.

Coding drift across the study window

ICD-10 replaced ICD-9 on a rolling schedule, so a registry that spans that change contains both. A generated recode maps one scheme and treats the other as invalid, or worse, silently collapses non-matching codes into a missing category. Name the transition in the prompt or it will not be handled.

Missing that is not missing at random

dropna() is the reflex, and it is often wrong. A transferred patient, a death before the follow-up visit, and a genuine data-entry gap all look identical to pandas — its missing-data guide notes that NaN, NaT, and NA are sentinels and that equality tests against them return False, so pd.isna() is the only reliable detector. The sentinel tells you a value is absent. It does not tell you why, and the why decides whether you drop, impute, or model it.

Merge keys that are not unique

A patient identifier repeated across encounters turns a one-to-one join into a many-to-many expansion, and the row count grows without an error. Generated join code rarely asserts key uniqueness first.

The load, validate, clean, document pipeline

Four stages, each with a human gate.

Load. Read every column as string, inspect the raw values, then cast deliberately. Automatic type inference on a messy extract hides the coercions you most need to see.

Validate. Write assertions before any cleaning: identifier uniqueness where you expect it, dates inside the study window, categorical levels against a known list, physiological ranges on vitals and labs. A failed assertion is a finding, not an obstacle.

Clean. One transformation at a time, each line commented with the reason it exists. Batch rewrites are hard to review and harder to defend.

Document. Save session info and package versions next to the output, keep a data dictionary, and write a plain-language note for every row you dropped or value you imputed.

Prompts that surface the clinical traps

Generic "clean this dataframe" produces generic output. The prompts that name the trap do better:

• "This registry spans the ICD-9 to ICD-10 transition. Flag any diagnosis recode that assumes a single scheme."
• "Some missing outcomes are structural — transfer, or death before follow-up. Do not impute or drop until I classify each pattern."
• "List every assumption this join makes about the merge keys, then add an assertion for each."

The model is good at executing a named check. It is poor at knowing which checks a clinical dataset needs.

Before the cleaned file becomes results

A cleaned dataset carries decisions a reviewer will ask you to justify: how missingness was handled, how codes were harmonised, which records were excluded and why. Those decisions belong in the methods section, and they are the same class of gap covered in five failure modes of AI statistical code and the statistical assumption check workflow.

AI for Academic's peer-review stress test at aiforacademic.world reads a methods section against the reporting standard for your study type and flags the data-handling steps you did not describe. Free to start.