# Architecture > Status: built. Every milestone below is implemented; the package layout reflects the code. ## The problem CoDNA solves A company's knowledge is scattered across Slack threads, Google Docs, PDFs, and people's heads. Teams increasingly ask an AI agent instead of searching, but the agent has nowhere trustworthy to look — and if you give it access to everything, you have quietly destroyed your access controls. CoDNA is a knowledge base whose primary consumer is an AI agent, and whose primary constraint is that the agent must see exactly what the person it is acting for is allowed to see. Not more. ## Shape of the system ``` ┌──────────────────────────────┐ ┌────────────────────────┐ │ Claude · ChatGPT · Gemini │ │ Web UI (humans) │ │ any MCP client │ │ Svelte, embedded │ └──────────────┬───────────────┘ └───────────┬────────────┘ │ MCP (streamable HTTP) │ REST + cookies ▼ ▼ ┌───────────────────────────────────────────────────────────┐ │ codna — one static binary │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ transport: http handlers · mcp tools · embedded UI │ │ │ │ Both routes resolve to the same Actor. │ │ │ └──────────────────────┬──────────────────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ domain: documents · revisions · linkgraph · access │ │ │ │ suggestions · automations · ingestion · search │ │ │ └──────────┬──────────────────────────────┬───────────┘ │ │ │ │ │ │ ┌──────────▼───────────┐ ┌────────────▼───────────┐ │ │ │ outbox worker │ │ scheduler (cron) │ │ │ │ goroutine pool │ │ FOR UPDATE SKIP LOCKED │ │ │ └──────────┬───────────┘ └────────────┬───────────┘ │ └─────────────┼──────────────────────────────┼──────────────┘ │ │ ┌───────▼───────┐ ┌───────▼──────────┐ │ PostgreSQL │─────────────▶│ Typesense │ │ source of │ via outbox │ derived index │ │ truth │ │ (never truth) │ └───────────────┘ └──────────────────┘ ``` Three rules hold this together: 1. **One permission check.** The domain layer decides who can do what. The REST API and the MCP server are transport, not policy. A rule fixed once is fixed everywhere. Nobody is exempt from it: there are no administrators, and management — which runs the instance — can read nothing ([adr/0013](https://codna.me/docs/adr/0013-no-administrators.md)). 2. **Postgres is truth; everything else is derived.** Typesense can be dropped and rebuilt from Postgres at any time. It is never written to directly. 3. **One process, one binary.** API, MCP, workers, scheduler, and UI ship together. There is no separate queue, no sidecar, and no `node_modules` at runtime. ## The event spine Every mutation writes its change **and** an event row in the same transaction: ``` BEGIN UPDATE documents … INSERT document_revisions … INSERT events … ← same transaction COMMIT ``` A worker then drains `events` and fans out. This is the transactional outbox pattern, and it is the single most load-bearing decision in the design: | Consumer | What it does | |---|---| | Search indexer | Upserts or removes the document in Typesense | | Link resolver | Re-extracts links, resolves pending ones, marks stale and broken edges | | Automation runner | Matches enabled automations against the event and runs their scripts | | Notifier | Raises access, broken-link, and suggestion notifications | | *(seam)* AI workers | Not built. Server-side distillation plugs in here later as one more consumer. | **Why it matters.** Without the outbox, a write to Postgres followed by a write to Typesense can half-succeed: the document exists but is unsearchable, or worse, a revoked share is still visible in search results. With it, the event is committed atomically with the change, so a consumer that crashes simply retries — the index converges, it never silently diverges. It also makes consumers pluggable. The hybrid AI position — agents drive today, server-side workers maybe later — costs nothing precisely because "later" means *adding a consumer*, not restructuring the write path. Consumers must be idempotent: delivery is at-least-once, not exactly-once. A failed event is tried again after a wait that doubles from half a minute up to an hour, ten attempts in all -- about three hours, enough to ride out Typesense or the database being away -- before it becomes a dead letter for a person to look at. The event remembers which consumers already succeeded, and a retry runs only the rest, so a failing indexer does not make the notifier tell anyone twice. A consumer that panics fails its event like any other error, and the process stays up. Dispatch uses Postgres `LISTEN`/`NOTIFY` through pgx to wake the worker immediately, with a periodic poll as a floor so a missed notification delays work rather than losing it. Claiming an event takes two mechanisms, not one. `FOR UPDATE SKIP LOCKED` stops two workers racing on the same row during the claiming statement; a `claimed_at` lease stops the second worker picking it up again *after* that statement commits, which it otherwise would, since the row stays unprocessed until the consumer finishes. An expired lease returns the event to the pool, so a worker that dies mid-flight delays its events rather than stranding them. The lease suits work measured in seconds. Reading an imported export can take many minutes, so imports have a worker of their own beside the outbox's, claiming with the same two mechanisms but holding its lease by renewing it as it goes ([ingestion.md](https://codna.me/docs/ingestion.md#imports)). The pages it writes are ordinary events on the outbox, so the index, the link graph and folder shares hear of each one as they would of any other. ## Live updates A page someone has open hears when what they can see, or what waits on them, changes, and fetches it again: share a document with Dana and her open tab shows it within a second — the inbox badge, Home, the sidebar's open folders, the Inbox, a folder's page — without a reload. Taking a share away empties them the same way. ``` document_access ─┐ triggers pg_notify('codna_live', ┌─ hub (internal/live) ─┐ GET /api/me/live notifications ─┼──────────▶ 'access group:', ─────▶│ one LISTEN connection │────────────────▶ page link_suggestions ┘ 'inbox user:') │ per process │ event: changed └───────────────────────┘ {"what":"access"} ``` - **Postgres says what changed and for whom**, from row triggers (migration 00007), so every way a share or a notification is made — the API, MCP, automations, the command line, a quarantine — is heard without any of them having to remember to. A bulk share signals each principal once: Postgres drops a notification identical to one already sent in the same transaction. - **The hub** holds one connection of its own, outside the pool, and wakes the streams of the people a signal is for: a group's members, looked up then, or every stream for Everyone, which has no member rows. A stream folds whatever arrives before it writes into one event per kind. - **The stream says only that something changed**, `access` or `inbox`, never what. The page asks the API again as its person, and the API decides what they may see, so nothing here can leak a document. It holds no database connection; it ends after fifteen minutes and the browser reconnects, which checks the session again. - **The page in view is watched on the same stream** (`?watch=`, migration 00018): a new version saved, a deletion or restore, or a quarantine wakes only the streams watching that document, and each looks it up again as its person and says which version is current — `event: document`, `{"id":…,"version":N}` — or that it is gone for them. After gone it says nothing more of it, so a page someone may not read cannot be watched for when it changes. - **Nothing is queued, and nothing needs to be.** A signal missed — the hub reconnecting, a page between streams — costs a refresh a little later: the hub tells everyone to look again when it listens again, and a page looks again whenever its stream reopens or the tab comes back into view. A hidden tab lets go of its stream, so background tabs hold no connections. A page being read shows the new version. A page being written merges it into the text, in the browser (`web/src/lib/editor/merge.ts`): both sides are measured from the version the edit began from, line by line and then, for lines both changed, word by word. What only one side changed is taken from that side, placed in the editor where it belongs so the cursor keeps its place, and kept out of the undo history so undoing one's own typing never takes out someone else's save. Changes that overlap or touch are not merged: the writer is told, before they press Done, and compares in the conflict panel. A save refused as stale is merged the same way once and sent again, and a draft left on the device from an older version is merged with what was saved since. Nothing is ever saved over someone else's version without either merging it or the writer choosing to. The stream is server-sent events, and a proxy must not buffer it. CoDNA sends `X-Accel-Buffering: no`, which turns buffering off in nginx, and a comment every 25 seconds, inside nginx's default 60-second read timeout; Caddy passes event streams through as they are. ## Automations: events and schedules Automations consume the same event stream, plus a clock. They are user-authored JavaScript, written in practice by an AI agent on request. ``` events ──────────▶ ┌──────────────────┐ │ automation │──▶ QuickJS in wazero ──▶ proposed actions scheduler ───────▶ │ matcher │ │ (cron, Postgres) └──────────────────┘ ▼ host validates each action against the automation OWNER's permissions │ ▼ applied in one transaction ``` Triggers fall into four families — document lifecycle, access changes, link-graph changes, and schedule; an ingested document arrives as `document.created` like any other. A scheduled automation has **no subject document**, which is why the capability API includes bounded reads (`codna.find`, `codna.get`) and not only actions. Those reads run with the automation owner's visibility, so a script cannot read past its author. The scheduler is Postgres-backed rather than in-process: due automations are claimed with `FOR UPDATE SKIP LOCKED`, so running several replicas neither double-fires nor drops a tick. Full contract in [automations.md](https://codna.me/docs/automations.md). ## Where the AI lives CoDNA does not call an LLM. Agents connect over MCP and do the thinking: ``` person: "build a wiki page from the onboarding PDFs I uploaded" agent: search(space=raw, query="onboarding") get_document(...) ×4 create_document(space=wiki, ...) suggest_links(...) → person approves ``` This means no provider API key is required to run CoDNA, no inference cost, and no opinion about which model is best. Every MCP-capable client works on day one. The seam for server-side workers exists (above) for the cases agents cannot cover: bulk backfill of a 10,000-document import, or scheduled re-distillation. That is deliberately not in the first milestones. ## Package layout ``` cmd/codna/ serve · migrate · bootstrap · reindex · ingest · import · export · people · groups · tokens · status · events (running the instance) internal/ domain/ core types and the permission policy — pure, no I/O store/ sqlc-generated queries; hand-written SQL lives in db/queries/ search/ Typesense collections, indexing, ACL-filtered queries indexer/ the outbox consumer that keeps search in step with Postgres markdown/ goldmark parsing, frontmatter, link extraction and editing document/ create, update, move, delete, revisions, diffs, backlinks linkgraph/ extraction, resolution, state transitions and their events sharing/ grants, revokes, share suggestions, access events people/ people, groups, and handing on what someone owned; management's to change notify/ notifications and link decisions automation/ registration, runner, outbox consumer, scheduler sandbox/ QuickJS on wazero: limits, host calls, the JS prelude outbox/ event worker, consumer dispatch, cascade depth live/ telling open pages what changed for their person: LISTEN, fan-out maintenance/ retention jobs, under an advisory lock status/ operational counts and dead-letter handling, for management audit/ the management audit: what management did, append-only breakglass/ finding, unsharing and quarantining documents by metadata, never content ingest/ file conversion, raw document creation storage/ blob storage for ingested originals and pages' files attachment/ pages' images and PDFs: upload, sniffing, naming, reading through the page export/ the markdown tree out, and back in auth/ passwords, sessions, OIDC verification, service tokens api/ HTTP handlers mcpserver/ MCP tool definitions and bearer auth testdb/ testidp/ per-package test databases; a signing identity provider for tests db/ migrations/ goose SQL migrations queries/ SQL for sqlc web/ Svelte 5 + SvelteKit static; built to web/build, embedded via embed.FS ``` `internal/domain` holds the permission policy as pure functions with no database access, so it can be exhaustively tested against a truth table rather than inferred from integration tests. ## Technology choices Full rationale, with rejected alternatives, in [adr/](https://codna.me/docs/adr/README.md). | Layer | Choice | Why | |---|---|---| | Language | Go 1.27+ | One static binary, no runtime dependencies, goroutines suit the outbox and scheduler | | Database | PostgreSQL 17 | Recursive CTEs for link traversal, GIN for tags, JSONB for frontmatter, `SKIP LOCKED` for the scheduler | | Queries | sqlc | Hand-written SQL, generated types. The hard queries here are recursive CTEs — an ORM would fight them | | Migrations | goose | Plain SQL files, reviewable in a pull request | | Driver | pgx v5 | `LISTEN`/`NOTIFY`, native array and JSONB handling | | Search | Typesense 29 | Typo tolerance, faceting, `string[]` filters for ACL, optional vector search | | Markdown | goldmark | CommonMark-compliant with a real AST — link extraction must not be a regex over prose | | Frontmatter | `goccy/go-yaml` | Round-trips ordering and comments | | Diff | `hexops/gotextdiff` | Unified diffs computed on read | | Sandbox | QuickJS on wazero (`fastschema/qjs`) | CGO-free. wazero gives a syscall-free substrate; QuickJS enforces memory and execution-time limits internally | | Cron | `adhocore/gronx` | Expression parsing only — schedule state stays in Postgres so replicas stay correct | | IDs | UUIDv7 | Time-sortable, so index locality and "recent" queries stay cheap | | Web UI | Svelte 5 + Vite, `adapter-static` | Compiles to a static tree that `embed.FS` takes directly; no runtime Node | | Tests | stdlib `testing` + testcontainers-go | Real Postgres and Typesense — the interesting behaviour is in SQL and index filters | ## Deployment One deployment serves one company. Users, documents, and groups carry no org scoping — which keeps every query and every permission check meaningfully simpler. A hosted offering would run one deployment per customer. ``` podman run -e DATABASE_URL=… -e TYPESENSE_URL=… ghcr.io/torniker/codna serve ``` Retrofitting multi-tenancy later means touching essentially every query. That is understood and accepted; see [adr/0002](https://codna.me/docs/adr/0002-single-org-per-deployment.md). ### Deployment strategies Three, all running the same binary — full guides in ../deploy/README.md. | Strategy | Artifacts | |---|---| | Compose | `compose.yaml`, CoDNA behind an `app` profile | | Kubernetes | `deploy/kubernetes/` manifests + kustomization, or `deploy/helm/codna/` | | VM / bare metal | `deploy/systemd/` — sandboxed unit plus env template | Builds target OCI rather than a vendor: `Containerfile` and `compose.yaml` work with podman, docker, buildah, and kaniko unchanged. See [adr/0011](https://codna.me/docs/adr/0011-oci-containers-and-deployment.md). Two operational facts worth knowing before you deploy: - **Migrations are embedded in the binary.** `codna migrate up` applies the schema from a copy compiled in at build time, so the schema can never disagree with the code running against it, and no deployment ships migration files. Run it as a Job or an `ExecStartPre` — not as an initContainer on a multi-replica Deployment, since goose takes no lock by default. - **The `local` storage driver pins you to one replica.** Ingested originals need a writable volume and ReadWriteOnce cannot be shared. CoDNA is otherwise replica-safe — the outbox worker and scheduler both claim work with `FOR UPDATE SKIP LOCKED` — so scaling out needs only ReadWriteMany or an object-store driver. ## Milestones | Milestone | Scope | |---|---| | **M1** | Data model and technology spec *(this document set)* ✅ | | **M2** | Schema, domain, Typesense indexing, outbox worker, sharing ✅ | | **M3** | REST API and auth — sessions for the UI, OIDC tokens for agents ✅ | | **M4** | MCP server — search, read, write, share-with-suggestions ✅ | | **M5** | Web UI — read, edit, history and diffs, sharing, search ✅ | | **M6** | Raw ingestion — uploads, paste, bulk load, six converters ✅ | | **M7** | Automations — sandbox, triggers, scheduler, run records ✅ | | **M8** | Groups and accounts, export and import, CI and release ✅ | | **M9** | Manageable by a machine — service tokens, instance identity, status, dead letters, export over HTTP ✅ | | **M10** | Web UI, second pass — folder tree and quick open, CodeMirror editor with drafts and conflict resolution, history restore, move and restore, share by tag, related links, automations editor, accounts, settings, admin; one token set for both themes, a phone layout, keyboard shortcuts ✅ | | **M11** | Web UI, third pass — one New and one search, an account menu, spaces listed once they hold something; a page for connecting an AI agent (`mcp` on `/api/instance`); a home with Get started and Continue reading; Edit, Share and More on a document, its title once; a share dialog that names people and groups, and upgrades a reader; a link picker and writing from source material; an inbox that keeps or restores a link in place and counts the news (`kept_at`, `/api/me/inbox-counts`) ✅ | | **M12** | Web UI, fourth pass — a folder opens as a page of its own, its folders and then its pages (`exact=1` on `GET /api/documents`), and the tree keeps its place; writing that looks as it reads: markdown shown only around the cursor, a click into the text edits there and Done goes back to the same place, a toolbar over a selection and the usual keys, a slash menu for blocks, a checkbox ticked while reading saved as a version of its own. The editor changes only the characters it must ([ADR 0006](https://codna.me/docs/adr/0006-inline-markdown-links.md)) and loads only for someone who can write ✅ | | **M13** | Images and files in pages ([ADR 0015](https://codna.me/docs/adr/0015-page-owned-files.md)) — pasted, dropped or chosen from `/image`, kept in `.files/` beside the page and linked as plain markdown; drawn while writing and reading; served through the page's own permission with a sandboxing policy; exported beside the page and imported back; `get_file` hands an agent the image ✅ | | Later | Server-side AI workers, outbound webhooks, OCR, connectors, multi-org, git mirror | Auth is deliberately absent until M3. M2 exercises permissions through an `Actor` value constructed directly in tests, which is enough to prove the rules without committing to an auth mechanism.