The CHIRPdb dataset

What you have

Read access to a running PostgreSQL database, the source report PDFs, and the paperwork to make sense of both. No access to CHIRP's application, pipeline or APIs; everything you need is in the four pieces below.

Working with the database

It is ordinary PostgreSQL - three things worth knowing before you query:

• Every table is readable. All tables allow public read; there is nothing to unlock first.
• Vector columns need pgvector. passage_embeddings.embedding and shield_code_embeddings.embedding are vector(1024) columns; a client without pgvector reads them as text. The match_shield_codes(passage_id, strategy, limit) function does the cosine search server-side, so you do not need one.
• tsvector columns are generated. The full-text search columns are computed automatically from their source text and are read-only - query them for search, don't write them.

What the dataset is

Each published report (a PDF, from MAIB or NTSB) is broken into sentence-level text plus the report's own stated incident particulars. MAIB reports are additionally linked to structured incident data - occurrences, vessels, affected persons - drawn from the MAIB Data Portal spreadsheets; NTSB has no equivalent spreadsheet source, so that layer is MAIB-only. An analysis pipeline then extracts findings from the text: safety issues, recommendations, and named entities with every mention resolved. Every finding is traceable back to the exact sentences of the source report. CHIRP's SHIELD taxonomy - 113 causal-factor codes in 24 categories under four layers - is loaded, and every passage is embedded so codes can be suggested, but no code has yet been matched to a finding.

documents.jurisdiction separates the two corpora: 'UK' for the 796 MAIB reports, 'US' for the 456 NTSB reports - 1,252 documents in all. The pipeline has been run over all 1,252, so the database is populated end to end; what is absent is listed under What is not included, and the one substantive gap is that missing SHIELD link.

What is actually populated

The database is populated end to end - every pipeline stage has run over the whole corpus. Five tables are empty, for three different reasons; know which before you build on them.

How the data was produced

The data flows through pipeline stages, and the tables mirror them. Provenance is preserved as a chain of links, not copied data: any finding traces to its passage, the passage to its sentences, each sentence to its document and position.

How the tables fit together

Two incident-data sources are kept apart and linked, not merged: the report PDF's own account lives in document_particulars / document_vessels (both corpora); the MAIB Data Portal spreadsheet's account in occurrences / vessels / affected_persons (MAIB only); the two join through a nullable documents.occurrence_id FK - several reports can point at one occurrence (e.g. an interim then a final). Tree-shaped classifications (event type, ship type, place on board, injury, deviation) are normalised into taxonomy_terms. The full entity-relationship diagram:

− 100% + Fit width Actual size Full screen Scroll to pan, or hold Ctrl/Cmd and scroll to zoom.

The data dictionary is authoritative for table and column definitions; the diagram is the map, the dictionary the legend.

Where each value comes from - and how far to trust it

This is the point of the dictionary. The database shows you structure; it cannot tell you which values are source fact, which a model produced, and which a human has since confirmed. Every column is labelled with one of the five below - the first two both mean "the source said so", and differ only in which source: a report PDF, or the MAIB spreadsheet.

A slashed label - Pipeline / AI-U - marks a column whose extractor is deterministic with a model fallback.

Two rules run throughout. Row-level verification: tables with an is_verified flag hold AI-generated content whose status varies per row - treat is_verified = true as AI-V, everything else AI-U. Sentence text is extracted by a vision LLM reading each page image, with deterministic passes on top, so it is AI-U, qualified by the parent documents.status (verified = analyst-confirmed).

Known limitations

Read these before drawing conclusions from the data.

• Two jurisdictions, not a global sample. Reports come from the UK MAIB (796 documents, published 2015-2026) and the US NTSB (456 documents, 2010-2026) - patterns describe UK and US waters and the vessels those two bodies investigate, not global shipping. Split or control by documents.jurisdiction: the corpora differ in body, report format, date span and field coverage, so pooling them silently mixes populations.
• Different reporting thresholds either side. Each body investigates only accidents meeting its own criteria, and those criteria differ. This is not a random sample; minor and unreported occurrences are absent by construction, so incidence rates cannot be estimated from it, and MAIB/NTSB counts are not comparable as rates.
• Spreadsheet incident data is MAIB-only, and sparsely linked. occurrences, vessels, affected_persons and affected_persons_ppe come from the MAIB Data Portal and cover UK incidents alone; no NTSB document has an occurrence_id. Even on the MAIB side the report↔occurrence link is partial: 58 of the 796 MAIB documents carry one. Treat the occurrence tables as an incident dataset in their own right, not as an enrichment of every report.
• The report corpus and the spreadsheet barely overlap. occurrence_id links only 58 reports to an occurrence - just 61 of the 9,063 occurrences carry a publication_number at all, because MAIB publishes a report for a small fraction of the incidents it records. Any analysis joining report text to spreadsheet incident attributes runs on those 58 documents.
• Nothing has been verified by an analyst. Verification is explicit and per-row - is_verified = true on safety_issues, recommendations and passage_shield_codes - and no row anywhere carries it; all 1,252 documents sit at status = 'extracted', none at verified. narrative_entity_mentions.confidence is the extraction's own self-report, not a measured accuracy.
• No SHIELD code has been applied. The taxonomy is loaded in full (113 codes in 24 categories across 4 layers) and every passage is embedded, so the machinery for suggesting codes exists, but the link from a finding to a causal-factor code (passage_shield_codes) has not been made. This is the main analytical gap in the dataset.
• Findings follow the report's own structure. A safety issue is classified as contributory or addressed only where the report itself classified it: is_contributory is set on 2,975 of 11,254 rows and is_addressed on 3,035; recommendations.reference_number on 2,414 of 4,127 and addressee on 3,159. Absence is a property of the source reports, not a gap in extraction - do not impute the unfilled rows.
• Report-stated particulars are sparser for NTSB. document_particulars and document_vessels exist for every document, but fields are filled only where the report states them, and NTSB reports state fewer. Of 456 NTSB / 796 MAIB rows: accident_date 198 / 796, accident_location 208 / 731, severity 71 / 305, accident_type 130 / 279, loss_of_life 33 / 183; document_vessels.port_of_origin and destination are empty for NTSB (286 / 262 for MAIB), while vessel_name is better covered for NTSB (400 / 299).
• authors is not canonicalised. Its 22 rows carry PDF-metadata name variants of the two bodies (MAIB, Marine Accident Investigation Branch, www.maib.gov.uk; NTSB, National Transportation Safety Board, the typo NSTB) plus individual people named as a PDF's author. Use documents.jurisdiction to identify the publishing body, not authors.name.
• Text extraction is AI-driven and imperfect. Each page was rendered to an image and read by a vision LLM, with deterministic passes on top. Expect residual artefacts: merged/split sentences, headers absorbed into text, occasional transcription slips, and - where documents.extraction_fallback = true - a degraded result for some pages. 201 documents are flagged that way (182 MAIB, 19 NTSB).
• Sentence text is AI-produced, not byte-exact. It is the model's reading of the page image - a faithful-but-not-verbatim transcription. status = 'verified' documents are analyst-confirmed; extracted ones are not. As delivered all 1,252 documents sit at extracted - none has been analyst-verified. The source PDFs ship with the dataset, so anything that matters can be checked against the original page.
• Mixed categorical vocabularies. Report-stated fields (document_particulars.accident_type / severity, document_vessels.vessel_type) are free text - normalise before counting, and note MAIB and NTSB word them differently, so the free text is doubly heterogeneous. Identifiers differ too: publication_number is a MAIB Publication_No for UK reports and an NTSB report number (MAB-, MIR-, MAR-) for US ones, and document_type is a GOV.UK publication class that is null for every NTSB document. Spreadsheet classifications are normalised into taxonomy_terms and referenced by *_term_id. Coordinates are verbatim text, not validated geo types.

What is not included

Five tables are empty, for three different reasons:

• passage_shield_codes - the missing link. No matching pass has been run and no analyst has accepted a suggestion, though everything the match needs is present.
• analysis - a convenience table for free-form per-passage output, so a future pass has somewhere to write without a schema change. Nothing writes it today.
• legislation and safety_issue_legislation - attaching legislation to a safety issue is a designed but unbuilt feature. author_identifiers is likewise unpopulated.

Also absent: pipeline observability data (LLM traces, extraction logs) and any application authentication data.

Row counts

As of 28 August 2026. Seeded vocabularies are loaded reference data.

Tandem Creative Dev · prepared for Zig Zag AI / CHIRP, for the Alan Turing Institute Data Study Group · 2026
Documentation conventions: data dictionary follows DAMA-DMBOK; this overview follows the Datasheets for Datasets framing.