Ingestion

Status: built. Converters, the pipeline, the REST endpoints, the ingest_text MCP tool, the upload page, imports of whole exports from the web, recordings transcribed through a transcription service, and codna import all exist. OCR and live connectors remain deferred.

Ingestion gets existing material into CoDNA: uploads, pasted text, scraped pages, whole exports from another tool, recordings of meetings. Everything it produces lands in raw as markdown, with the original file kept untouched.

Ingestion does not produce knowledge. It produces evidence. Turning evidence into a wiki is an agent's job, and a separate step.

Pipeline

upload / paste / URL
      │
      ▼
 store the original, byte-for-byte      ──▶  storage_key
      │
      ▼
 ingestions row  (status='pending')
      │
      ▼
 detect type  (gabriel-vasile/mimetype — content sniffing, not the file extension)
      │
      ▼
 convert to markdown
      │
      ▼
 create a `raw` document                ──▶  documents.source_ingestion_id
      │
      ▼
 events: ingestion.completed  →  indexer · automations · notifier

The original is stored before conversion is attempted. A conversion that fails, or one that succeeds badly, must never mean asking someone to find and re-upload a file.

Converters

• .md and .txt pass through as they are. Frontmatter is lifted into frontmatter_extra, not left in the body.
• .html and web pages are converted by JohannesKaufmann/html-to-markdown/v2. The title is taken from <title>.
• .docx is read with archive/zip and encoding/xml, over word/document.xml. Headings, lists and tables survive; styling, images and footnotes do not.
• .pdf is read by ledongthuc/pdf. Text only. See the caveat below.
• .csv is read by encoding/csv into a markdown table, so the contents are searchable prose rather than a blob.
• .json is pretty-printed in a fenced block, by the standard library.
• Audio and video become recordings: kept, filed, and transcribed in the background by a transcription service, not converted here.
• Anything else is refused, with the original stored: a format we cannot read today is one we may read next release.

.docx is parsed here rather than through a library. It is sixty lines of token-walking, and this sits on the ingestion path for material people may not have another copy of — the available Go options are young enough that a parsing bug would be ours to discover in production. The walk is token-based rather than struct unmarshalling because a document body interleaves paragraphs and tables, and unmarshalling into separate slices silently reorders the document, hoisting every table to the end.

Conversion is deliberately partial. Raw is evidence of what a document said; the original is kept for anything more.

The PDF caveat, stated plainly

Go's document-extraction ecosystem is weaker than Python's, and PDF is where the gap shows. There is no pure-Go equivalent of PyMuPDF or pdfminer. Expect:

• Good results on text-native PDFs with a simple single-column layout.
• Degraded results on multi-column layouts, heavy tables, and unusual font encodings.
• Nothing at all from scanned PDFs — there is no OCR in the pipeline. A PDF that yields far less text than its page count suggests returns a warning rather than a silently empty document, which is the failure worth catching: it looks ingested, it is searchable, and it contains nothing.

Three things make this acceptable rather than a blocker:

1. Originals are always retained, so re-running a better converter later is a background job, not a migration.
2. ingestions.status='failed' and a low-yield conversion are both visible, so a bad extraction is noticed rather than silently becoming bad knowledge.
3. An external extractor — Apache Tika or unstructured in a sidecar — can be configured for deployments that need it. That trades away the single-binary story, which is why it is opt-in rather than the default.

The converter is an interface with one method. Replacing the PDF implementation touches one package.

Storage

Originals live outside Postgres:

storage_key = raw/2026/09/018f3c2a-….pdf

resolved against CODNA_STORAGE_PATH by the local driver. storage.Open picks the driver from CODNA_STORAGE_DRIVER, so an S3 implementation is one case in one function.

Date-partitioned, keyed by the ingestion's UUIDv7, with the original filename preserved in the ingestions row rather than in the path. Two people uploading handbook.pdf must not collide, and filenames from the wild contain things no filesystem should be asked to hold.

Local disk is the default. An S3-compatible backend is the obvious second implementation; the interface is get/put/delete over an opaque key.

The raw document

One raw document per ingestion. Its fields come from:

• space: raw.
• path: YYYY/MM, or a path the uploader chose.
• slug: the filename or title, slugified and deduplicated.
• title: the document's title if the format has one, otherwise the filename.
• content: the converted markdown.
• frontmatter_extra: source, original_filename, mime_type, ingested_at and byte_size.
• source_ingestion_id: the ingestions row.
• tags: whatever the uploader supplied.

Raw documents are not writable — not by their uploader, not by anyone (permissions.md). Correcting raw material means ingesting a corrected version, which leaves both and an audit trail. If raw could be edited, nothing derived from it could be trusted to reflect what was actually received.

Slugs are deduplicated with a numeric suffix. Ingesting the same file twice produces two documents, deliberately: they may differ, and deciding they are the same is a judgment call, not a hash comparison.

Surfaces

• Web. /upload takes files dragged in, or content pasted with its source; audio and video go up in pieces. /import takes a whole export as one zip. /record records a meeting.
• REST.
  • POST /api/ingest/upload takes files, as multipart with a repeatable file field.
  • POST /api/ingest/text takes content pasted with its source.
  • GET /api/me/ingestions lists the caller's ingestions, recordings still arriving included.
  • POST /api/imports starts an import, GET /api/imports lists them and GET /api/imports/{id} shows one.
  • /api/recordings takes recordings in pieces; see Recordings.
• MCP. ingest_text files given material. list_ingestions answers "where did my upload go?", and for a recording, how far its transcript has got.
• Command line. codna import --user <email> <dir> reads a markdown tree on the host.

A batch upload reports per file: one unreadable PDF among twenty does not discard the other nineteen, and the response lists what landed and what did not. That report is returned even when every file failed — flattening it into a single error would drop exactly the per-file reasons the uploader needs.

The ingestion history is per-person and includes failures. A failed ingestion produces no document, so without it a file that could not be converted would look indistinguishable from one that vanished.

Imports

A company starting on CoDNA already knows a great deal, in another tool. An import brings a whole export in from the web: upload one zip, get its pages as a folder in raw, with the export's folders kept and the links between its pages pointing at where those pages landed. Raw, because what arrives from outside is evidence: what a team concludes from it is written in wiki, linked back, as with any upload (adr/0016).

Export	What it looks like
Notion, "Markdown & CSV"	A page per file, its title and id in the name; subpages in a folder of the same name; a database as a CSV and a folder of row pages; a large export as parts zipped into one download
Confluence, HTML space export	Pages side by side, the tree in each page's breadcrumbs; attachments filed under numbers
Obsidian, or any folder of markdown	Folders and pages as they are, wikilinks included
Google Drive	Files downloaded from Drive: Google Docs arrive as Word files, PDFs as themselves

What an import does

1. Keeps the archive. It is stored byte-for-byte as an ingestion before anything is read, and it is what a later import of the same archive reads again.
2. Checks it while the uploader waits: that it is a zip, and within the limits below. A wrong file is refused at once, with the reason.
3. Reads it in the background. An export of thousands of pages takes longer than a request should wait, and longer than the outbox's lease. A worker in every replica claims queued imports under a row lock and holds each with a lease it renews as it goes; a worker that dies leaves its import to the next. The page shows how many pages are done, of how many.
4. Lays the pages out as the export had them, beneath one folder: raw:imports/<name> unless the person chose another. An export that is one page with everything beneath it -- a Notion workspace's top page, a Confluence space's home -- makes that page the folder's introduction (index). Every address is planned before any page is written, so a link can reach a page not yet written, and an address someone else's page holds is stepped around (-2).
5. Cleans the names. Notion's ids come off titles and folders, Confluence's space name comes off titles, and a page's title is its first heading where the file name lost something (a colon, the end of a long title). In a vault the file name is the title.
6. Points links at their pages. A relative link to another page of the export becomes a relative link to that page's address (linking.md). So does a link pasted from the browser that names a page by its id (a notion.so link, a Confluence page link), an Obsidian wikilink, and a Confluence link by title. Links to the web are left alone, and nothing in code is touched. A link to nothing the export holds keeps its words, loses its markup, and is listed.
7. Files each page through ingestion, private to the importer as every new page is. Each page carries its provenance: which export, its id there, where in the archive it sat. A file a converter reads -- a PDF attached to a Notion page, a text file attached in Confluence -- becomes a page of its own beneath the page it was attached to.
8. Keeps each page's images as the page's own files, written with it (below).
9. Reports. Pages imported, updated and unchanged; images kept; images and attachments not brought over, with why and the page that used each; links left as words; pages that could not be read; and the export's own files it passed over. One bad page never stops the rest.

The import page then offers to share the whole folder, standing, through a folder share (adr/0014): its pages now, and those a later import adds.

Importing again

A newer export imported into the same folder brings the pages up to date. Pages are matched by their identity in the source (a Notion page id, a Confluence page id, a file's path), recorded as each page is written, so a page is found wherever it has been moved since. A page whose text changed gets a new version, with the earlier one kept in its history; one whose text did not gets nothing. A page deleted here since stays deleted, and a page gone from the export stays as it is; the report lists both. A link to a page gone from the newer export still reaches it. A different folder makes a fresh copy. Only the importer's own earlier imports count: another person's pages in the same folder are theirs.

Images and attachments

An image a page shows, and an image it links to, comes over as one of the page's own files (files.md, adr/0015): it is stored in <slug>.files/ beside the page, and the page's link is pointed there. Notion's Page 1b2c…/architecture.png becomes deploying.files/architecture.png; a Confluence attachment filed as attachments/65540/65545.png becomes deploying.files/pipeline.png, under the name it had in Confluence; an Obsidian ![[diagram.png|320]] becomes ![diagram.png](home.files/diagram.png). An image used by two pages is a file of each, since a file belongs to one page and has its permission.

A file arrives with the version of the page that names it. Its bytes are stored first, and its record is written in the same transaction as that version, so no version of a raw page ever names a file it does not hold. A page that fails to be written takes its files' bytes with it.

Each file is named as an upload would be: the export's name, slugified, with the extension of what the bytes are (Screen Shot.png becomes screen-shot.png). Two images on one page whose names come out the same are two files (screen-shot.png, screen-shot-2.png); the same picture shown twice is one.

What a page cannot hold is not brought over. It stays a note of its name where it was (*[image: flow.svg]*), a link to it keeps its words, and the report says why, with the page that used it:

• an SVG, which can carry script;
• an image of another kind (BMP, TIFF, HEIC) or one whose bytes are not an image at all;
• a file larger than 20 MB, or an empty one;
• an image the export does not include, such as an export made without its files;
• any other kind of file a page links to, a spreadsheet or an archive;
• an image no page shows.

An image shown from the web (https://…) is left showing from there, and listed, since nothing keeps it: it shows for as long as its host keeps it. A PDF attached to a page is not a file of the page but a page of its own, with its text, as above.

A re-import keeps a page's images in step with the export. An image whose bytes did not change is the file the page already holds, so a page whose text and pictures are the same gets no version. An image that changed is a new file beside the old one (architecture-2.png), named by the new version, while the earlier version still shows the old one, as files never change. The archive is kept, so pages imported before images came over get them by importing the same archive into the same folder again: each page with a picture gains a version that shows it.

Limits

An archive is untrusted. A few kilobytes can claim to hold terabytes, name a file ../../etc/passwd, or hold a million empty entries.

• The archive: CODNA_MAX_IMPORT, 512 MiB by default, spooled to disk rather than held in memory.
• Each file read out of it: CODNA_MAX_UPLOAD. A larger one is skipped and reported.
• 50,000 files at most, counting the parts of a split export.
• 4 GiB read out of one archive in all, and no file over a megabyte that expands more than 200 times: a zip bomb is reported as a page that could not be read, and the rest goes on.
• A name that is absolute or climbs out with .. is refused, not cleaned; links inside the archive are not followed; names are made valid UTF-8. Nothing is ever written to disk under a name from the archive.
• What operating systems and editors leave behind (__MACOSX, .DS_Store, .obsidian, a vault's trash) is passed over without a word.

A proxy in front of CoDNA has to accept a request as large as CODNA_MAX_IMPORT on /api/imports.

The seam for connectors

Formats are one interface: lay the archive out as a tree of pages keyed by the source's own ids, and read one page as markdown. Everything after that -- addresses, links, writing, re-imports, the report -- is shared. A live connector, Google Drive kept in sync, is the same shape with a different start: its pages come from Drive's API rather than an archive, keyed by Drive's file ids, and an import into the same folder is already what keeping it in sync means.

Recordings

A meeting is evidence like any document, and usually the only record of what was decided. A recording — a meeting someone recorded online and uploads, or one recorded in the room on a phone — becomes a raw page with its transcript: who spoke, when, and what they said. An agent distils it into wiki like any other raw page.

CoDNA does not transcribe. It sends the recording to a transcription service at CODNA_TRANSCRIBE_URL, and writes the page from what the service answers. The contract between them is Transcription; the reasoning is adr/0017. Without a service, recordings are kept and their pages say they were not transcribed.

What happens to a recording

1. It arrives in pieces. A file is sent 8 MiB at a time, and a meeting recorded in the browser five seconds at a time. Pieces are numbered from 0 and taken in order; the same piece sent again is taken as already here, and one sent ahead of a missing one is answered with the number wanted. So any length of recording gets through a proxy that takes 64 MB, and an upload cut off part-way carries on where it stopped. Only audio and video are taken, told from the bytes of the first piece.
2. It is put together and kept as the original of its page, on disk, never in memory, and the pieces are let go.
3. Its page is filed at once, in raw:YYYY/MM like any upload, saying it is being transcribed. The recording plays from the page.
4. It is sent to the service, by a worker that takes one step at a time under a lease, as imports are taken: send it, then ask how it stands every few seconds, then every half minute. A service that is down is waited out for about an hour; one that refuses the recording ends it.
5. The transcript becomes the page's next version: a paragraph for each turn, opening with the speaker and the time it started, **Speaker 1** (12:40) …. A speaker's consecutive words join until two minutes have passed. The frontmatter records the recording and the transcript:
   • original_filename, mime_type, byte_size;
   • source: upload, browser or meeting;
   • transcription: pending, done, failed or none;
   • length (1:02:34) and length_seconds, language, engine, speakers;
   • kept_for_training: whether the service kept a copy of this recording, and transcription_notice when it said something to show with it.
6. A failure gets a version saying why, the ingestion is marked failed with the reason, and the person who made the recording can have it transcribed again from its page. So can a recording kept untranscribed, once a service is configured.

The page is filed before the transcript exists because a recording should have an address from the start: to share, to play, to link to, and to say what went wrong if it does.

Recording in the browser

New → Record a meeting records from the microphone, which is asked for only when Record is pressed. MediaRecorder hands over a piece every five seconds; each is written to the browser's IndexedDB before it is sent, and dropped from the device only once the server has it. Offline, pieces wait on the device. A closed tab, a crash or a flat battery loses at most the last few seconds:

• pieces still on the device are offered on /record the next time it is opened, to send the rest;
• pieces already on the server are transcribed even if the device never comes back. A browser recording that stops arriving is finished from what arrived a day after its last piece, or at once from /record on another device.

The screen is kept on while recording, since a phone that locks may stop the page.

Telling the people recorded

Whether the service keeps recordings is its policy, read from it before recording. On codna.me, staff decide it per instance; the people in the room did not. So while the service says recordings are kept, Record shows its notice and waits for whoever records to confirm they told the room, and Upload shows the notice beside audio and video. The transcript then records whether that recording was kept.

Meetings

New → Start a meeting holds an online meeting through a meetings service at CODNA_MEET_URL (Meetings, adr/0018). Its page is filed in raw as the meeting starts, with Join, the guest link and Invite people; once the call is over, the meeting's mix becomes the page's original, and each person's track goes to the transcription service under their name, so the transcript says who said what. Its frontmatter adds source: meeting, participants, meeting_started and meeting_ended.

Limits

• A recording: CODNA_MAX_RECORDING, 4 GiB by default. A meeting's mix and each of its tracks are held to it too.
• A piece: 32 MiB, which is what a proxy in front of CoDNA has to accept on /api/recordings.
• An upload left part-sent is thrown away after a week; a recording from the browser is finished a day after its last piece.

REST

• GET /api/transcription: whether recordings are transcribed here, and the service's retained and notice.
• POST /api/recordings starts one: filename, title, source (upload or browser), and for a file its size.
• PUT /api/recordings/{id}/pieces/{n}: piece n, as the body. A 409 carries next, the piece wanted.
• POST /api/recordings/{id}/finish with pieces, how many were sent.
• GET /api/recordings/{id}, and GET /api/recordings?unfinished=true.
• POST /api/recordings/{id}/transcribe: transcribe it again.
• DELETE /api/recordings/{id}: throw away one still arriving.
• GET /api/documents/{id}/recording: the recording itself, to whoever can read its page, with ranges.
• Meetings have their own: see Meetings.

A recording can also be uploaded whole through POST /api/ingest/upload, within CODNA_MAX_UPLOAD.

Raw becomes wiki

CoDNA does not do this. An agent does:

person: "build an onboarding page from the IT handbook I just uploaded"

agent:  search(space='raw', query='IT handbook')
        get_document('raw:2026/09/it-handbook')
        create_document(space='wiki', path='engineering/onboarding', …,
                        content: '…extracted from [the IT handbook](../../raw/2026/09/it-handbook.md)')
        suggest_links(...)

The link back into raw is an ordinary markdown link, so provenance is a normal graph edge: backlinks from a raw document show everything distilled out of it, and share suggestions walk wiki → raw like any other edge.

Nothing enforces that a wiki document cites raw material. It is a convention the MCP tool descriptions encourage, and it is worth keeping — "where did this claim come from?" is the question a knowledge base lives or dies on.

Failure

status='failed', error='…'
  → the original is still stored
  → notification to the uploader
  → retry after fixing, or download the original and handle it manually

Conversion failure never loses the file and never produces a half-converted document. Either a raw document is created from a complete conversion, or the ingestion is marked failed with the original intact.

Deferred

• OCR for scanned PDFs and images. Needs an external engine; the natural home is the same optional sidecar as heavy PDF extraction.
• URL ingestion with crawl. Single-page fetch is straightforward; following links is a different feature with its own scope and politeness rules.
• Live connectors — Google Drive kept in sync, Slack; recordings fetched from Meet, Teams and Zoom. Each is an ingestion source plus an auth story, and a recording connector feeds the same recording path. Exports from Notion, Confluence, Obsidian and Drive are imported today (Imports), and a connector would feed the same importer.
• Deduplication by content hash across ingestions, surfaced as a suggestion rather than enforced.
• Incremental re-ingestion of single uploads — uploading a changed file as a new version of the same raw document rather than a second one. Imports already do this for a whole export (Importing again).