Data and storage

References configuration

GNOSI_DATA_DIR/config/references.json owns the bibliographic table designation, its explicit disabled state and linked-attachment settings. Source-tree copies are migrated explicitly by scripts/migrate-reference-config.py, never by an API request or implicit startup copy. The migrator preserves the original and all unknown fields, verifies exact UTF-8 JSON bytes, publishes without replacing another file and keeps a private recovery journal. Startup refuses unmigrated legacy configuration before database upgrades and background workers.

Ownership map

Data	Durable owner	Rebuild or recovery rule
Page content and front matter	Markdown vault	Back up and version as ordinary files.
Attachments and library files	Active vault	Preserve relative or portable references.
Internal page metadata	Vault .gnosi sidecars	Migrate with the page; hide implementation-only fields from authored content.
Page and wikilink indexes	Local data caches	Rebuild from vault; partial scans must not overwrite complete caches.
Users, workspaces, memberships, vault access, PATs, shares	Management SQLite	Back up as local application state; never cloud-sync the live database.
Mail, reader, notification, annotation, and execution indexes	Local SQLite	Domain-dependent; recover from providers or source data where possible.
OAuth tokens and integration secrets	Local-data secrets or OS credential store	Reconnect per machine if lost; never copy to a shared vault.
Agent checkpoints	Local data	Per-instance execution memory, not vault content.

Vault format

A page is a Markdown file with YAML front matter. Stable page identifiers allow links and relations to survive title changes. Human-visible links use wikilink syntax; attachments and file-valued properties use portable paths or structured metadata rather than machine-specific absolute paths.

Database-style views are projections over pages and registries. They do not replace Markdown with an opaque relational store. View definitions, schema metadata, formulas, rollups, relations, and presentation state are resolved by the vault service layer.

Required plugin fields

Plugin field contracts persist config.plugin_roles beside the immutable property ID. Processes resolve semantic roles to IDs and read current names or historical aliases through the field resolver. Schema replacement and field PATCH requests reject removal, ID replacement, type changes and relation retargeting with HTTP 409; renames remain available. Server-owned bindings survive editor round trips.

LLM Wiki activation receives the active interface locale (Catalan, English, Spanish or French). Reconciliation repairs stale role IDs, binds existing fields, creates missing fields and seeds the idea and verification catalogs. Existing custom names and option labels remain stable. Required choices cannot be removed; renaming a choice updates its semantic mapping and existing records.

Idea type classifies Entity, Concept, Summary or Synthesis. Verification tracks Provisional, Verified, Needs review or Refuted separately from workflow status. Last reviewed records knowledge review, separately from arbitrary file edits. Generation writes the idea type, provisional verification, reading order and review date through the registered IDs. Lint reads the same review role; index ordering and resource processing stamps likewise resolve stable IDs.

Write concurrency

Page reads expose an ETag derived from the current representation. Mutating clients return the expected ETag; mismatches reject stale writes instead of silently overwriting a concurrent change. Atomic-write helpers replace files only after the new representation is complete.

Rename operations depend on the wikilink index to rewrite inbound links. A rename therefore crosses page identity, file naming, registry metadata, sidecars, and link indexes and must be treated as a coordinated operation.

Management database

SQLAlchemy models represent:

erDiagram
    USER ||--o{ MEMBERSHIP : joins
    WORKSPACE ||--o{ MEMBERSHIP : contains
    WORKSPACE ||--o{ VAULT : registers
    VAULT ||--o{ VAULT_ACCESS : grants
    USER ||--o{ VAULT_ACCESS : receives
    USER ||--o{ API_TOKEN : owns
    USER ||--o{ SHARE_LINK : creates

All management-database models inherit one typed SQLAlchemy DeclarativeBase. Engine and session factories are initialized atomically and return concrete Engine and Session types; the same metadata and table names remain authoritative for Alembic and existing SQLite installations.

Before workers start, the schema coordinator resolves the management database, every dynamic vault and each durable first-party auxiliary store. Independent Alembic revision lines recognize exact reviewed 2.x structural fingerprints, create verified backups and apply forward-only upgrades. Unknown or drifted schemas abort without an application schema migration. Derived caches and externally owned databases remain outside Gnosi's migration heads.

Startup opens an existing owned database for read/write access under its migration lock so SQLite can roll back an interrupted transaction before inspecting the committed schema. This native crash recovery preserves committed data and does not authorize unknown schema upgrades. Read-only audits keep read-only connections and leave any hot rollback journal untouched.

The scoped academic_index.sqlite3 files belong to the literature_index family. OAI records and oai_sync_state are durable; the FTS virtual table is rebuildable but remains in the same reviewed migration because its row IDs are kept in lockstep with the durable records. Runtime connections set only SQLite operational pragmas and never execute schema DDL.

Only PAT hashes and a recognizable prefix are persisted. Public share tokens are opaque identifiers whose rows retain creator, vault, permission, expiry, and revocation state.

Local-data isolation

GNOSI_DATA_DIR points to the per-instance root. The path resolver creates cache, system, checkpoint, log, audio, output, backup, and secret directories. Docker maps this to /data; native defaults follow the operating system's application-data convention. GNOSI_LOCAL_DATA remains a deprecated 3.x alias.

SQLite files must not be placed on OneDrive, iCloud Drive, Dropbox, or another file-sync layer. File synchronization does not provide SQLite locking semantics and can corrupt or fork the database.

Cloud-backed vaults

File-provider adapters separate ordinary filesystem behavior from hydration and availability. Reads catch transient per-file errors and continue when a partial response is meaningful. A partial scan is marked and must never be saved as a complete cache. Native macOS hydration uses a GUI-session action because a LaunchAgent process may receive EDEADLK for online-only content. The hydration runtime is provider-neutral: OneDrive, iCloud Drive, Google Drive, Nextcloud and Dropbox have separate adapters and configuration prefixes. An unknown service mounted under macOS ~/Library/CloudStorage uses the generic fileprovider adapter. Ordinary mounted or fully synchronized folders use local-filesystem behavior.

Configuration ownership

Configuration is deep-merged from base parameters and the applicable user or active-vault .gnosi/params.yaml. Environment values override deployment paths and a small set of bootstrap behavior. Credentials are references into local secret storage, not raw values embedded in the vault configuration.

Process variables have priority over Gnosi's local .env. A shared environment file is loaded only when GNOSI_SHARED_ENV_FILE explicitly names it and remains read-only to the application. UI-managed credentials use the operating-system credential store, with an encrypted fallback under GNOSI_DATA_DIR/secrets.