Open source · Python · Apache-2.0

Clinical data at scale, with its metadata intact.

clinfolio stores CDISC datasets in Parquet, Avro and HDF5, and keeps labels, codelists, origins and lineage attached through every filter, join and derivation.

$ pip install clinfolio (coming soon)

Pre-release. The name is reserved; the first version is in development.

See how it works Follow on GitHub

advs.filter(PARAMCD == "SYSBP")advs/ · 48.2 M rows · Parquet

USUBJIDUnique Subject Identifier

101-001

101-002

102-004

PARAMCDParameter Code

SYSBP

PULSE

SYSBP

DIABP

SYSBP

AVISITAnalysis Visit

Week 8

Week 4

Week 8

AVALAnalysis Value

128

141

133

BASEBaseline Value

139

146

137

CHGChange from Baseline

−11

−3

−5

−4

Rows drop out. Labels, codelists and origins stay on every column; green marks a derived variable.

EEG_C3 · 256 Hz · HDF5 → linked to SDTM

Three formats, one metadata model

Each format stores the same CDISC metadata natively, so any Arrow-aware tool can read it and conversion between formats is lossless.

Parquet

For analysis at scale

Columnar SDTM and ADaM, partitioned by study, subject or parameter. Queries read only the columns and row groups they need.

Avro

For streaming and ingestion

Row-based records for EDC, eCOA and device feeds, validated on arrival, with schema evolution tied to define.xml versions.

HDF5

For signals and dense arrays

PSG, actigraphy, ECG and glucose waveforms stored beside tabular SDTM, chunked and compressed, with links back to subject records.

Process a billion rows without losing the define

clinfolio adds a CDISC layer on top of polars and DuckDB. Work stays lazy and out of core, and every derived column records where it came from.

import clinfolio as cf

advs = cf.scan("advs/")          # lazy, nothing loaded
adsl = cf.scan("adsl.parquet")

out = (advs
  .filter(cf.col("PARAMCD") == "SYSBP")
  .merge_adsl(adsl, ["TRT01A", "SAFFL"])
  .derive_change(base_flag="ABLFL")
  .window_visits(spec="windows.yaml"))

out.write_parquet("advs_sysbp/", partition_by=["AVISITN"])
out.lineage().to_define("define.xml")

Lineage for CHG

VS.VSSTRESNCollected · Vital Signs

ADVS.AVALPredecessor · Analysis Value

ADVS.BASEDerived · AVAL where ABLFL = "Y"

ADVS.CHGDerived · AVAL − BASE, post-baseline visits

What happens to metadata in each operation

Propagation rules are written down and tested, so the output always has a complete, correct specification.

Operation	Labels and types	Keys and sort order	Origin and lineage
filter	Kept unchanged	Kept; row count updated	Filter condition recorded
merge_adsl	Carried from ADSL	Left dataset's keys kept	Predecessor: ADSL.variable
derive_*	Set from the method	Unchanged	Derived, with the method text
union	Checked for conflicts; stops on a mismatch	Re-sorted by keys	Both sources recorded
write_*	Written natively to the format	Row groups sorted by keys	Content hash added

From raw signal to analysis value, traceably

Device data rarely fits in a SAS dataset. clinfolio keeps waveforms in HDF5 and links every derived measure back to the samples it came from.

1 · Store

Raw waveform

EDF or device export imported with sampling rate, units and start time.

2 · Epoch

30-second epochs

Resampled, time-aligned to the dosing clock, and scored or summarized.

3 · Link

SDTM and ADaM rows

Summary measures become records with a link back to the source epochs.

USUBJID PARAMCD AVAL SRC 101-001 SLPEFF 87.2 EEG_C3[0:960]

Roadmap

The first release focuses on one thing: CDISC metadata that survives processing.

Parquet and metadataMetadata model, Parquet codec, XPT and Dataset-JSON bridges
Lazy frameMetadata propagation and core CDISC operations
Validate and lineageChecks that scale, plus column lineage
Avro and streamingValidated ingestion and incremental rebuilds
HDF5 and signalsWaveforms, epochs and PSG examples
BenchmarksResults at 10⁸–10⁹ rows and a proposal to CDISC