# Ingestion > Status: built. Converters, the pipeline, the REST endpoints, the `ingest_text` MCP tool, the > upload page, imports of whole exports from the web, recordings transcribed through a transcription > service, and `codna import` all exist. OCR and live connectors remain deferred. Ingestion gets existing material into CoDNA: uploads, pasted text, scraped pages, whole exports from another tool, recordings of meetings. Everything it produces lands in `raw` as markdown, with the original file kept untouched. Ingestion does **not** produce knowledge. It produces evidence. Turning evidence into a wiki is an agent's job, and a separate step. ## Pipeline ``` upload / paste / URL │ ▼ store the original, byte-for-byte ──▶ storage_key │ ▼ ingestions row (status='pending') │ ▼ detect type (gabriel-vasile/mimetype — content sniffing, not the file extension) │ ▼ convert to markdown │ ▼ create a `raw` document ──▶ documents.source_ingestion_id │ ▼ events: ingestion.completed → indexer · automations · notifier ``` The original is stored **before** conversion is attempted. A conversion that fails, or one that succeeds badly, must never mean asking someone to find and re-upload a file. ## Converters - **`.md` and `.txt`** pass through as they are. Frontmatter is lifted into `frontmatter_extra`, not left in the body. - **`.html` and web pages** are converted by `JohannesKaufmann/html-to-markdown/v2`. The title is taken from ``. - **`.docx`** is read with `archive/zip` and `encoding/xml`, over `word/document.xml`. Headings, lists and tables survive; styling, images and footnotes do not. - **`.pdf`** is read by `ledongthuc/pdf`. Text only. See the caveat below. - **`.csv`** is read by `encoding/csv` into a markdown table, so the contents are searchable prose rather than a blob. - **`.json`** is pretty-printed in a fenced block, by the standard library. - **Audio and video** become [recordings](#recordings): kept, filed, and transcribed in the background by a transcription service, not converted here. - **Anything else** is refused, with the original stored: a format we cannot read today is one we may read next release. `.docx` is parsed here rather than through a library. It is sixty lines of token-walking, and this sits on the ingestion path for material people may not have another copy of — the available Go options are young enough that a parsing bug would be ours to discover in production. The walk is token-based rather than struct unmarshalling because a document body interleaves paragraphs and tables, and unmarshalling into separate slices silently reorders the document, hoisting every table to the end. Conversion is deliberately partial. Raw is evidence of what a document *said*; the original is kept for anything more. ### The PDF caveat, stated plainly Go's document-extraction ecosystem is weaker than Python's, and PDF is where the gap shows. There is no pure-Go equivalent of PyMuPDF or pdfminer. Expect: - Good results on text-native PDFs with a simple single-column layout. - Degraded results on multi-column layouts, heavy tables, and unusual font encodings. - **Nothing at all from scanned PDFs** — there is no OCR in the pipeline. A PDF that yields far less text than its page count suggests returns a warning rather than a silently empty document, which is the failure worth catching: it looks ingested, it is searchable, and it contains nothing. Three things make this acceptable rather than a blocker: 1. Originals are always retained, so re-running a better converter later is a background job, not a migration. 2. `ingestions.status='failed'` and a low-yield conversion are both visible, so a bad extraction is noticed rather than silently becoming bad knowledge. 3. An external extractor — Apache Tika or `unstructured` in a sidecar — can be configured for deployments that need it. That trades away the single-binary story, which is why it is opt-in rather than the default. The converter is an interface with one method. Replacing the PDF implementation touches one package. ## Storage Originals live outside Postgres: ``` storage_key = raw/2026/09/018f3c2a-….pdf ``` resolved against `CODNA_STORAGE_PATH` by the `local` driver. `storage.Open` picks the driver from `CODNA_STORAGE_DRIVER`, so an S3 implementation is one case in one function. Date-partitioned, keyed by the ingestion's UUIDv7, with the original filename preserved in the `ingestions` row rather than in the path. Two people uploading `handbook.pdf` must not collide, and filenames from the wild contain things no filesystem should be asked to hold. Local disk is the default. An S3-compatible backend is the obvious second implementation; the interface is get/put/delete over an opaque key. ## The raw document One `raw` document per ingestion. Its fields come from: - **`space`**: `raw`. - **`path`**: `YYYY/MM`, or a path the uploader chose. - **`slug`**: the filename or title, slugified and deduplicated. - **`title`**: the document's title if the format has one, otherwise the filename. - **`content`**: the converted markdown. - **`frontmatter_extra`**: `source`, `original_filename`, `mime_type`, `ingested_at` and `byte_size`. - **`source_ingestion_id`**: the `ingestions` row. - **`tags`**: whatever the uploader supplied. Raw documents are **not writable** — not by their uploader, not by anyone ([permissions.md](https://codna.me/docs/permissions.md#the-policy)). Correcting raw material means ingesting a corrected version, which leaves both and an audit trail. If raw could be edited, nothing derived from it could be trusted to reflect what was actually received. Slugs are deduplicated with a numeric suffix. Ingesting the same file twice produces two documents, deliberately: they may differ, and deciding they are the same is a judgment call, not a hash comparison. ## Surfaces - **Web.** `/upload` takes files dragged in, or content pasted with its source; audio and video go up in pieces. `/import` takes a whole export as one zip. `/record` records a meeting. - **REST.** - `POST /api/ingest/upload` takes files, as multipart with a repeatable `file` field. - `POST /api/ingest/text` takes content pasted with its source. - `GET /api/me/ingestions` lists the caller's ingestions, recordings still arriving included. - `POST /api/imports` starts an import, `GET /api/imports` lists them and `GET /api/imports/{id}` shows one. - `/api/recordings` takes recordings in pieces; see [Recordings](#recordings). - **MCP.** `ingest_text` files given material. `list_ingestions` answers "where did my upload go?", and for a recording, how far its transcript has got. - **Command line.** `codna import --user <email> <dir>` reads a markdown tree on the host. A batch upload reports per file: one unreadable PDF among twenty does not discard the other nineteen, and the response lists what landed and what did not. That report is returned even when every file failed — flattening it into a single error would drop exactly the per-file reasons the uploader needs. The ingestion history is per-person and includes failures. A failed ingestion produces no document, so without it a file that could not be converted would look indistinguishable from one that vanished. ## Imports A company starting on CoDNA already knows a great deal, in another tool. An import brings a whole export in from the web: upload one zip, get its pages as a folder in `raw`, with the export's folders kept and the links between its pages pointing at where those pages landed. Raw, because what arrives from outside is evidence: what a team concludes from it is written in `wiki`, linked back, as with any upload ([adr/0016](https://codna.me/docs/adr/0016-imports-into-raw.md)). | Export | What it looks like | |---|---| | Notion, "Markdown & CSV" | A page per file, its title and id in the name; subpages in a folder of the same name; a database as a CSV and a folder of row pages; a large export as parts zipped into one download | | Confluence, HTML space export | Pages side by side, the tree in each page's breadcrumbs; attachments filed under numbers | | Obsidian, or any folder of markdown | Folders and pages as they are, wikilinks included | | Google Drive | Files downloaded from Drive: Google Docs arrive as Word files, PDFs as themselves | ### What an import does 1. **Keeps the archive.** It is stored byte-for-byte as an ingestion before anything is read, and it is what a later import of the same archive reads again. 2. **Checks it** while the uploader waits: that it is a zip, and within the limits below. A wrong file is refused at once, with the reason. 3. **Reads it in the background.** An export of thousands of pages takes longer than a request should wait, and longer than the outbox's lease. A worker in every replica claims queued imports under a row lock and holds each with a lease it renews as it goes; a worker that dies leaves its import to the next. The page shows how many pages are done, of how many. 4. **Lays the pages out** as the export had them, beneath one folder: `raw:imports/<name>` unless the person chose another. An export that is one page with everything beneath it -- a Notion workspace's top page, a Confluence space's home -- makes that page the folder's introduction (`index`). Every address is planned before any page is written, so a link can reach a page not yet written, and an address someone else's page holds is stepped around (`-2`). 5. **Cleans the names.** Notion's ids come off titles and folders, Confluence's space name comes off titles, and a page's title is its first heading where the file name lost something (a colon, the end of a long title). In a vault the file name is the title. 6. **Points links at their pages.** A relative link to another page of the export becomes a relative link to that page's address ([linking.md](https://codna.me/docs/linking.md)). So does a link pasted from the browser that names a page by its id (a notion.so link, a Confluence page link), an Obsidian wikilink, and a Confluence link by title. Links to the web are left alone, and nothing in code is touched. A link to nothing the export holds keeps its words, loses its markup, and is listed. 7. **Files each page** through ingestion, private to the importer as every new page is. Each page carries its provenance: which export, its id there, where in the archive it sat. A file a converter reads -- a PDF attached to a Notion page, a text file attached in Confluence -- becomes a page of its own beneath the page it was attached to. 8. **Keeps each page's images** as the page's own files, written with it (below). 9. **Reports.** Pages imported, updated and unchanged; images kept; images and attachments not brought over, with why and the page that used each; links left as words; pages that could not be read; and the export's own files it passed over. One bad page never stops the rest. The import page then offers to share the whole folder, standing, through a folder share ([adr/0014](https://codna.me/docs/adr/0014-standing-folder-shares.md)): its pages now, and those a later import adds. ### Importing again A newer export imported into the same folder brings the pages up to date. Pages are matched by their identity in the source (a Notion page id, a Confluence page id, a file's path), recorded as each page is written, so a page is found wherever it has been moved since. A page whose text changed gets a new version, with the earlier one kept in its history; one whose text did not gets nothing. A page deleted here since stays deleted, and a page gone from the export stays as it is; the report lists both. A link to a page gone from the newer export still reaches it. A different folder makes a fresh copy. Only the importer's own earlier imports count: another person's pages in the same folder are theirs. ### Images and attachments An image a page shows, and an image it links to, comes over as one of the page's own files ([files.md](https://codna.me/docs/files.md), [adr/0015](https://codna.me/docs/adr/0015-page-owned-files.md)): it is stored in `<slug>.files/` beside the page, and the page's link is pointed there. Notion's `Page 1b2c…/architecture.png` becomes `deploying.files/architecture.png`; a Confluence attachment filed as `attachments/65540/65545.png` becomes `deploying.files/pipeline.png`, under the name it had in Confluence; an Obsidian `![[diagram.png|320]]` becomes `![diagram.png](home.files/diagram.png)`. An image used by two pages is a file of each, since a file belongs to one page and has its permission. A file arrives with the version of the page that names it. Its bytes are stored first, and its record is written in the same transaction as that version, so no version of a raw page ever names a file it does not hold. A page that fails to be written takes its files' bytes with it. Each file is named as an upload would be: the export's name, slugified, with the extension of what the bytes are (`Screen Shot.png` becomes `screen-shot.png`). Two images on one page whose names come out the same are two files (`screen-shot.png`, `screen-shot-2.png`); the same picture shown twice is one. What a page cannot hold is not brought over. It stays a note of its name where it was (`*[image: flow.svg]*`), a link to it keeps its words, and the report says why, with the page that used it: - an SVG, which can carry script; - an image of another kind (BMP, TIFF, HEIC) or one whose bytes are not an image at all; - a file larger than 20 MB, or an empty one; - an image the export does not include, such as an export made without its files; - any other kind of file a page links to, a spreadsheet or an archive; - an image no page shows. An image shown from the web (`https://…`) is left showing from there, and listed, since nothing keeps it: it shows for as long as its host keeps it. A PDF attached to a page is not a file of the page but a page of its own, with its text, as above. A re-import keeps a page's images in step with the export. An image whose bytes did not change is the file the page already holds, so a page whose text and pictures are the same gets no version. An image that changed is a new file beside the old one (`architecture-2.png`), named by the new version, while the earlier version still shows the old one, as files never change. The archive is kept, so pages imported before images came over get them by importing the same archive into the same folder again: each page with a picture gains a version that shows it. ### Limits An archive is untrusted. A few kilobytes can claim to hold terabytes, name a file `../../etc/passwd`, or hold a million empty entries. - The archive: `CODNA_MAX_IMPORT`, 512 MiB by default, spooled to disk rather than held in memory. - Each file read out of it: `CODNA_MAX_UPLOAD`. A larger one is skipped and reported. - 50,000 files at most, counting the parts of a split export. - 4 GiB read out of one archive in all, and no file over a megabyte that expands more than 200 times: a zip bomb is reported as a page that could not be read, and the rest goes on. - A name that is absolute or climbs out with `..` is refused, not cleaned; links inside the archive are not followed; names are made valid UTF-8. Nothing is ever written to disk under a name from the archive. - What operating systems and editors leave behind (`__MACOSX`, `.DS_Store`, `.obsidian`, a vault's trash) is passed over without a word. A proxy in front of CoDNA has to accept a request as large as `CODNA_MAX_IMPORT` on `/api/imports`. ### The seam for connectors Formats are one interface: lay the archive out as a tree of pages keyed by the source's own ids, and read one page as markdown. Everything after that -- addresses, links, writing, re-imports, the report -- is shared. A live connector, Google Drive kept in sync, is the same shape with a different start: its pages come from Drive's API rather than an archive, keyed by Drive's file ids, and an import into the same folder is already what keeping it in sync means. ## Recordings A meeting is evidence like any document, and usually the only record of what was decided. A recording — a meeting someone recorded online and uploads, or one recorded in the room on a phone — becomes a raw page with its transcript: who spoke, when, and what they said. An agent distils it into wiki like any other raw page. CoDNA does not transcribe. It sends the recording to a **transcription service** at `CODNA_TRANSCRIBE_URL`, and writes the page from what the service answers. The contract between them is [Transcription](https://codna.me/docs/transcription.md); the reasoning is [adr/0017](https://codna.me/docs/adr/0017-recordings-become-raw-transcripts.md). Without a service, recordings are kept and their pages say they were not transcribed. ### What happens to a recording 1. **It arrives in pieces.** A file is sent 8 MiB at a time, and a meeting recorded in the browser five seconds at a time. Pieces are numbered from 0 and taken in order; the same piece sent again is taken as already here, and one sent ahead of a missing one is answered with the number wanted. So any length of recording gets through a proxy that takes 64 MB, and an upload cut off part-way carries on where it stopped. Only audio and video are taken, told from the bytes of the first piece. 2. **It is put together and kept** as the original of its page, on disk, never in memory, and the pieces are let go. 3. **Its page is filed at once**, in `raw:YYYY/MM` like any upload, saying it is being transcribed. The recording plays from the page. 4. **It is sent to the service**, by a worker that takes one step at a time under a lease, as imports are taken: send it, then ask how it stands every few seconds, then every half minute. A service that is down is waited out for about an hour; one that refuses the recording ends it. 5. **The transcript becomes the page's next version**: a paragraph for each turn, opening with the speaker and the time it started, `**Speaker 1** (12:40) …`. A speaker's consecutive words join until two minutes have passed. The frontmatter records the recording and the transcript: - `original_filename`, `mime_type`, `byte_size`; - `source`: `upload`, `browser` or `meeting`; - `transcription`: `pending`, `done`, `failed` or `none`; - `length` (`1:02:34`) and `length_seconds`, `language`, `engine`, `speakers`; - `kept_for_training`: whether the service kept a copy of this recording, and `transcription_notice` when it said something to show with it. 6. **A failure gets a version saying why**, the ingestion is marked failed with the reason, and the person who made the recording can have it transcribed again from its page. So can a recording kept untranscribed, once a service is configured. The page is filed before the transcript exists because a recording should have an address from the start: to share, to play, to link to, and to say what went wrong if it does. ### Recording in the browser **New → Record a meeting** records from the microphone, which is asked for only when Record is pressed. MediaRecorder hands over a piece every five seconds; each is written to the browser's IndexedDB before it is sent, and dropped from the device only once the server has it. Offline, pieces wait on the device. A closed tab, a crash or a flat battery loses at most the last few seconds: - pieces still on the device are offered on `/record` the next time it is opened, to send the rest; - pieces already on the server are transcribed even if the device never comes back. A browser recording that stops arriving is finished from what arrived a day after its last piece, or at once from `/record` on another device. The screen is kept on while recording, since a phone that locks may stop the page. ### Telling the people recorded Whether the service keeps recordings is its policy, read from it before recording. On codna.me, staff decide it per instance; the people in the room did not. So while the service says recordings are kept, Record shows its notice and waits for whoever records to confirm they told the room, and Upload shows the notice beside audio and video. The transcript then records whether that recording was kept. ### Meetings **New → Start a meeting** holds an online meeting through a meetings service at `CODNA_MEET_URL` ([Meetings](https://codna.me/docs/meetings.md), [adr/0018](https://codna.me/docs/adr/0018-meetings-recorded-per-person.md)). Its page is filed in raw as the meeting starts, with Join, the guest link and Invite people; once the call is over, the meeting's mix becomes the page's original, and each person's track goes to the transcription service under their name, so the transcript says who said what. Its frontmatter adds `source: meeting`, `participants`, `meeting_started` and `meeting_ended`. ### Limits - A recording: `CODNA_MAX_RECORDING`, 4 GiB by default. A meeting's mix and each of its tracks are held to it too. - A piece: 32 MiB, which is what a proxy in front of CoDNA has to accept on `/api/recordings`. - An upload left part-sent is thrown away after a week; a recording from the browser is finished a day after its last piece. ### REST - `GET /api/transcription`: whether recordings are transcribed here, and the service's `retained` and `notice`. - `POST /api/recordings` starts one: `filename`, `title`, `source` (`upload` or `browser`), and for a file its `size`. - `PUT /api/recordings/{id}/pieces/{n}`: piece n, as the body. A `409` carries `next`, the piece wanted. - `POST /api/recordings/{id}/finish` with `pieces`, how many were sent. - `GET /api/recordings/{id}`, and `GET /api/recordings?unfinished=true`. - `POST /api/recordings/{id}/transcribe`: transcribe it again. - `DELETE /api/recordings/{id}`: throw away one still arriving. - `GET /api/documents/{id}/recording`: the recording itself, to whoever can read its page, with ranges. - Meetings have their own: see [Meetings](https://codna.me/docs/meetings.md#rest). A recording can also be uploaded whole through `POST /api/ingest/upload`, within `CODNA_MAX_UPLOAD`. ## Raw becomes wiki CoDNA does not do this. An agent does: ``` person: "build an onboarding page from the IT handbook I just uploaded" agent: search(space='raw', query='IT handbook') get_document('raw:2026/09/it-handbook') create_document(space='wiki', path='engineering/onboarding', …, content: '…extracted from [the IT handbook](../../raw/2026/09/it-handbook.md)') suggest_links(...) ``` The link back into `raw` is an ordinary markdown link, so provenance is a normal graph edge: backlinks from a raw document show everything distilled out of it, and share suggestions walk `wiki → raw` like any other edge. Nothing enforces that a wiki document cites raw material. It is a convention the MCP tool descriptions encourage, and it is worth keeping — *"where did this claim come from?"* is the question a knowledge base lives or dies on. ## Failure ``` status='failed', error='…' → the original is still stored → notification to the uploader → retry after fixing, or download the original and handle it manually ``` Conversion failure never loses the file and never produces a half-converted document. Either a raw document is created from a complete conversion, or the ingestion is marked failed with the original intact. ## Deferred - **OCR** for scanned PDFs and images. Needs an external engine; the natural home is the same optional sidecar as heavy PDF extraction. - **URL ingestion with crawl.** Single-page fetch is straightforward; following links is a different feature with its own scope and politeness rules. - **Live connectors** — Google Drive kept in sync, Slack; recordings fetched from Meet, Teams and Zoom. Each is an ingestion source plus an auth story, and a recording connector feeds the same recording path. Exports from Notion, Confluence, Obsidian and Drive are imported today ([Imports](#imports)), and a connector would feed the same importer. - **Deduplication** by content hash across ingestions, surfaced as a suggestion rather than enforced. - **Incremental re-ingestion of single uploads** — uploading a changed file as a new version of the same raw document rather than a second one. Imports already do this for a whole export ([Importing again](#importing-again)).