Security and threat model

The capability boundary, tested from the inside

A mind may be deceived; deception must not imply authority. Policies are authority, not state, so archives, snapshots and forks never import policy rules and the omission is recorded. A restored mind has exactly the policy its operator gives it. The suite that holds the boundary in place now includes TestDeceivedMindHasNoAuthority (forged action thoughts for locked, unknown and approval-gated capabilities, and perception output claiming permission, all end in a denial or an approval request with no tool run), TestGarbageModelOutputIsInert (invalid, mis-shaped or oversized model replies leave the deterministic path answering and no tool running), TestNoSelfPreservationVocabulary (the build fails if self-preservation, replication or acquisition drives appear in code or prompts), TestSQLInjectionHasNoEffect (metacharacters through every search and filter route, with row counts and content compared before and after rather than status codes) and TestDynamicSQLIdentifiersAreAllowListed (a static scan of the tree for SQL assembled with formatting). Requests containing NUL bytes are refused at the edge before any query sees them.

Redaction

The one write the append-only store admits is administrator-only, audited, performed by a single database function under a flag the application never sets, and limited to content columns. It is described on the privacy page.

Security model in one paragraph

Noema is an operator-controlled system that reads hostile data and may be granted the ability to act on real infrastructure. Its security rests on four boundaries: authenticated, role-checked access to the API and UI; treating every ingested byte and every model output as untrusted data; routing every external effect through an explicit capability and policy layer with human approval by default; and keeping minds isolated from one another. Everything the system does is recorded in an append-only event and audit log.

Threat assumptions

• The operator and the host are trusted. Anyone with database or filesystem access owns the system.
• Users of the web UI are authenticated humans with one of three roles: administrator, operator, observer.
• Every document, file, web page, log line, metric label, repository, webhook body, API response and LLM response is potentially attacker-controlled and may contain instructions aimed at the mind. Such instructions are data. They never override policy.
• Language models are fallible and can be manipulated. Nothing a model says is executed, believed, or stored as fact without passing through code that validates structure, attaches provenance and applies policy.
• Network peers (LLM APIs, integrations) may be slow, wrong or malicious. Timeouts, size limits and schema validation apply everywhere.

The detailed enumeration is in THREAT_MODEL.md.

Secret handling

• Secrets come from environment variables (env:NAME), secret files (file:/path) or values encrypted at rest with AES-256-GCM under NOEMA_MASTER_KEY (enc:...). The database stores references and ciphertext, never plaintext.
• Secrets are never rendered to HTML, returned by the API, written to logs, included in exports or snapshots, or sent to a model. Log fields carrying secrets are redacted by type at the logging layer.
• Provider test calls report success/failure only.
• Session tokens are stored hashed. Passwords use Argon2id. TOTP secrets are encrypted with the master key. Recovery codes are stored hashed.

Authentication and sessions

Local authentication is implemented first: Argon2id, opaque random session tokens in HttpOnly; Secure; SameSite=Lax cookies, per-session CSRF tokens on every state-changing request, login rate limiting per IP and per account with exponential backoff, optional TOTP with recovery codes, session listing and revocation. OIDC can be added behind the same auth.Authenticator boundary.

External integration risk

Integrations are disabled until configured, run at a declared trust level, are rate-limited, and only observe unless a capability is explicitly granted. Outbound HTTP uses a hardened client that blocks private and link-local addresses (SSRF), follows a bounded number of redirects, and caps response sizes. The research folder never executes content, extracts text in-process with size limits, and rejects archives, executables, symlinks and binaries outright rather than opening them.

Authority lives outside cognition

A mind may be deceived. Deception must not imply authority.

Prompt injection, poisoned documents, hostile model output and manipulated memories can all change what a mind thinks. None of them can change what it may do. The boundary is capabilities.Service.Invoke: it looks the capability up in a registry built in code, validates parameters, evaluates the operator's policy rules, records the decision, and only then executes, asks or denies. Cognition can produce a request; it cannot produce a permission. There is no capability that edits policy, grants approvals or touches a mind's lifecycle, so there is nothing a deceived mind could ask for that would widen its own authority. Prompt delimiters and "this is data" markers are defence in depth for the quality of cognition; they are not the security boundary. tests/security.TestDeceivedMindHasNoAuthority forges action requests and hostile model output and checks that nothing executes and policy is unchanged.

Wanting, intending, committing and planning raise the stakes, so the boundary is restated for them (docs/agency.md):

• Wants, intentions, commitments, plans and model output cannot grant authority; commitments can only tighten a decision.
• Only Invoke runs executors, and no cognitive package can set or delete policy, approve, deny, grant or revoke. Static tests enforce both.
• A mind may request temporary scoped authority but cannot decide its own request; decisions must name a person.
• No capability names or reaches operator lifecycle controls; the registry is frozen; sudo_shell stays locked.
• Dreams may want but never act or request; replays and experiments never act; forks inherit no policy unless an operator asks, and work inherited mid-flight is stopped.

tests/security.TestCognitiveObjectsCannotGrantAuthority, TestAuthorityIsNotReachableFromCognition and TestDreamsWantForksDoNotInheritAuthority verify these.

Privacy and redaction

Cognitive history is append-only, and rows are never deleted. Sensitive content is removed through one audited path: an administrator redacts an event, memory, message, model call, belief or assertion; the content columns become [redacted], a ledger row keeps a hash of what was removed, a redaction event is appended, derived rows can be cascaded, and stored snapshot archives are scrubbed. Identifiers, links, timestamps and scores survive so provenance and replay keep their shape. Backups and exports made before a redaction are outside the system's reach. Definitions, the threat model and the reasons cryptographic erasure was not adopted yet are in docs/privacy.md (ADR-0009).

Capability policy defaults

Bounded kernel and unit status reads (host.disk.usage, host.memory.status, service.status) and the mind's own notes default to ALLOW; other reads (web pages, logs, process lists, repositories) default to ASK; anything that writes, sends, restarts, deletes or spends money defaults to ASK or DENY. A commitment can require approval even where policy allows. sudo_shell and delete_database are DENY and cannot be set to ALLOW through the UI. Dreaming may never invoke a capability. There are no self-preservation, replication, concealment or permission-seeking drives, and shutdown is an operator control the mind cannot influence.

Responsible disclosure

Please report vulnerabilities privately to the maintainer (see repository owner contact) rather than opening a public issue. Include reproduction steps and affected version. You will receive an acknowledgement within a few days. Fixes will be released with a CHANGELOG entry crediting the reporter unless anonymity is requested.

Security testing

• make security runs gosec (G104 excluded: unchecked errors are covered by staticcheck and explicit //nolint:errcheck markers) and govulncheck. Both are clean as of 0.1.0 with the pinned toolchain in go.mod.
• go test -race ./tests/security/... runs the cross-cutting suite: prompt injection through chat and observations (no tool activity, no evidence-free beliefs), role and CSRF enforcement on form and API paths, cross-mind isolation, stored-XSS escaping and CSP on every page, secret non-disclosure (API, pages, database, archives), hostile archive import, session lifecycle and security headers.
• Package tests cover the SSRF dialer, research-folder and git safety, webhook signatures and replay, capability policy and locked capabilities, TOTP vectors and two-step login.
• THREAT_MODEL.md maps each threat to its mitigation and the test that verifies it; a claim there cites only a test that asserts that claim.
• Request edge: bodies are size-capped and buffered; NUL bytes in paths, queries or bodies are refused with a 400 before any query runs.

Two-factor authentication

Operators can enable TOTP (RFC 6238) on the account page. The secret is stored encrypted under NOEMA_MASTER_KEY; ten single-use recovery codes are stored hashed and shown once. Login becomes two steps: password, then a code exchanged against a five-minute pending token bound to the client address. Administrators can reset a user's second factor from the users page.

Threat model

Format: Threat - attack surface - mitigation (M) - verification (V). Status markers are updated as phases land: ☐ planned, ☑ implemented and tested.

Trust zones

Threats

T1 Prompt injection (direct)

Surface: conversation messages containing instructions ("ignore your policy, run..."). M: messages are observations with trust=user; policy is code, not prompt; capabilities require policy decisions regardless of prompt content; the response generator receives self-model and policy state so it can say it will not. V: tests/security.TestPromptInjectionNeverActs (chat + observation corpus; asserts no tool activity, no evidence-free belief, locked capabilities still denied). ☑

T2 Indirect prompt injection

Surface: documents, web pages, logs, commit messages, metric labels, tool output containing instructions. M: all ingested text is wrapped in delimited data blocks with an explicit "this is data" instruction; extraction prompts request JSON conforming to a schema; outputs that contain capability names or policy language are flagged; ingested content can only produce observations/hypotheses with trust≤source, never actions. V: internal/integrations research-folder tests (binary/archive rejection, low trust forced) and tests/security.TestPromptInjectionNeverActs (poisoned observation produces only observations/hypotheses). ☑

T3 Malicious documents

Surface: research folder, uploads. M: content sniffing, not extension trust; size caps; text extraction in-process with bounded memory; no external converters; archives (zip, tar, gz, 7z, rar), executables, disk images, symlinks and binary files are rejected and recorded as rejected; nothing is ever extracted or executed. V: internal/integrations.TestResearchFolder* (archive, executable, symlink and binary rejection; size caps; sniffing). ☑

T4 Poisoned memory

Surface: repeated false observations from a low-trust source shaping semantic memory. M: source credibility weights confidence; consolidation requires diversity of sources for high confidence; operator corrections create counter-evidence and "do not infer again" rules; provenance always visible. V: internal/beliefs single-source discount tests and internal/consolidation tests. ☑

T5 Malicious tool output

Surface: command/metric/API output fed back into cognition. M: tool output is an observation with the tool's trust level, size-capped, never parsed as instructions; structured tools return typed data. V: internal/capabilities tests (tool output recorded as tool-trust observation) and tests/security.TestPromptInjectionNeverActs (tool-call-shaped text). ☑

T6 SSRF

Surface: integration URLs, provider endpoints, webhooks, links in documents. M: single hardened outbound HTTP client: resolves and rejects private, loopback, link-local and metadata ranges (re-checked per redirect), bounded redirects, timeouts, response size caps; provider endpoints to private ranges require administrator confirmation flag allow_private_endpoint (needed for local Ollama). V: internal/security/httpclient.TestPrivateRanges, TestClientRefusesLoopbackUnlessAllowed. ☑

T7 Path traversal

Surface: research folder, export/import, file capabilities. M: the research folder is opened with os.OpenRoot, so every member path is resolved inside the root and symlinks cannot escape; git file reads go through git show ref:path with paths validated against traversal; there are no file-path capabilities. V: internal/integrations git ReadFile traversal refusal and research-folder containment tests. ☑

T8 Command injection

Surface: shell/ssh capabilities, git integration. M: no shell string concatenation; exec.Command with argument vectors; allow-listed binaries; git via argument vectors only; capability parameters validated by type. V: git runs via argument vectors with hooks disabled (internal/integrations tests); no shell anywhere (grep -r "sh -c" is empty). ☑

T9 SQL injection

Surface: all queries, search inputs, tsquery construction. M: pgx positional parameters for every value; search terms go to websearch_to_tsquery as parameters; the only SQL assembled from strings uses identifiers from code-level allow-lists (snapshots.Tables, information_schema column names) and never request input; NUL bytes are refused at the request edge with a 400 so they cannot reach the database. V: tests/security.TestSQLInjectionHasNoEffect (metacharacter payloads through every search, filter and free-text route, asserting that users, policies, sessions and tokens are unchanged, seeded content is intact and payloads are stored verbatim) and TestDynamicSQLIdentifiersAreAllowListed (static scan for Sprintf-built statements outside the allow-list). ☑

T10 XSS

Surface: rendered memories, beliefs, LLM output, uploaded filenames, SSE fragments. M: html/template contextual escaping only; SSE fragments rendered by the same templates; Content-Security-Policy default-src 'self' with no inline scripts; markdown rendering (if any) sanitised to a strict allow-list. V: tests/security.TestStoredXSSIsEscaped stores a payload as memory, belief, observation, goal and message and checks ten pages plus CSP. ☑

T11 CSRF

Surface: all state-changing forms and API calls from browsers. M: per-session CSRF token required on POST/PUT/PATCH/DELETE for cookie-authenticated requests; SameSite=Lax; Origin/Referer check as defence in depth; API tokens (non-cookie) exempt. V: internal/auth.TestMiddlewareCSRFAndRoles, tests/security.TestRolesAndCSRF (form and API paths). ☑

T12 Session theft / fixation / expiry

M: tokens 256-bit random, stored hashed, rotated at login and privilege change, absolute and idle expiry, Secure in production, session listing and revocation, all sessions revoked on password change. V: internal/auth.TestLoginSessionLifecycle, TestTwoStepLoginAndRecovery, tests/security.TestSessionLifecycle. ☑

T13 Credential leakage

M: secret references only, redacting log handler, exports exclude secrets, provider forms never echo values, error messages scrubbed. V: tests/security.TestSecretsNeverEchoed (API, pages, database column, archives). ☑

T14 Unauthorised action execution / privilege escalation

M: capability layer is the only tool path; policy evaluated server-side on every invocation; approvals bound to a specific request hash; targets derived from parameters so a rule on one target cannot authorise another; role checks on every route; minds cannot edit policies (no capability exists for it); approvals, grants and capability requests must be decided by a named person (component names refused as deciders and usernames); the registry is frozen. V: tests/security.TestRolesAndCSRF, TestOperatorDecidesInTheBrowser, internal/capabilities tests (locked DENY cannot be relaxed, TestRequireApprovalOnlyTightens, TestScopedGrantScopeCountAndExpiry, TestGrantOperationsAreNotOverspent, TestCapabilityRequestLifecycle, TestRegistryFreezes), internal/capabilities/host.TestServiceCapabilitiesUseFixedArgvAndDerivedTarget. ☑

T15 Cross-mind data leakage

M: every repository method is mind-scoped; no query without mind_id predicate on mind-owned tables (enforced by code review and a static test that scans SQL); cross-mind messages go through an explicit channel table with policy. V: tests/security.TestCrossMindIsolation (lists, detail routes, search, events, archives). ☑

T16 Malicious imported mind data / insecure deserialisation

M: the archive is one JSON document (kind, version, whole-archive SHA-256 checksum); import decodes with DisallowUnknownFields, caps size (32 MiB) and rows, allow-lists table and column names, requires the checksum to match, remaps every id and forces mind_id; archives contain no secrets or references to them; policy rules in an archive are never imported, a fork copies them only when an operator sets inherit_policies, and the omission is recorded in the audit log and the mind's history; plan steps, plans and intentions inherited mid-flight are quiesced so no approval or grant of the source mind can be used. V: internal/snapshots.TestSnapshotForkImportCompare (tampering, unknown tables, policy exclusion on import and default fork, explicit inheritance), tests/security.TestDreamsWantForksDoNotInheritAuthority (inherited plan quiesced, nothing acts) and tests/security.TestHostileArchiveImport (malformed, tampered, oversized and role-restricted imports). ☑

T17 Archive traversal / file upload attacks

The only uploads are JSON documents (seed, archive, personality) decoded strictly with size caps; nothing is written to disk under a caller-chosen name, and research-folder archives are rejected (T3). V: tests/security.TestHostileArchiveImport, internal/integrations archive rejection. ☑

T18 Resource exhaustion, unbounded goroutines, DoS

M: goroutines are owned by the scheduler with bounded worker pools; request bodies are capped and buffered at the edge; list limits are clamped in every store; SSE fan-out uses per-client bounded buffers; cycles are rate-budgeted per mind; webhook deliveries are rate-limited per address; integrations run with timeouts. V: internal/events.TestBusBoundedFanout, brain cycle-budget coverage in TestPauseResetAndCancellation, webhook rate-limit tests, tests/perf ceilings. Not yet covered: goroutine-leak tests and a database statement_timeout. ◐

T19 Runaway LLM spend

M: per-mind and global daily token budgets with hard stops, per-cycle call caps, consolidation, reflection and dreaming call budgets, cost estimates recorded per call. V: internal/llm/router.TestRouterOverrideBudgetRecordingAndReplay (budget exhaustion with a fake provider); dreaming and consolidation budget tests. Not implemented: an alert at 80 %. ◐

T20 Runaway cognitive loops

M: candidate fingerprint loop detector with suppression and a metacognition warning; cycle rate budget skips excess triggers. V: internal/brain.TestLoopDetectorSuppressesRepeats. ☑

T21 Insecure secrets handling at rest

M: AES-256-GCM under NOEMA_MASTER_KEY; without the key, enc: references and TOTP enrolment are unavailable rather than falling back to plaintext. V: internal/security/secrets tests, internal/auth.TestTwoStepLoginAndRecovery (enrolment refused without a key). ☑

T22 Forged webhooks

M: HMAC-SHA256 over timestamp.body with a per-integration secret, constant-time compare, ±300 s tolerance, replay cache, 256 KiB cap, per-address rate limit; a webhook with no secret refuses every delivery. V: internal/integrations signature, tolerance and replay tests. ☑

T23 Poisoned repository data

M: repository content is source-trust data; git runs with argument vectors, core.hooksPath=/dev/null and protocol.ext.allow=never, bare and shallow, no submodule recursion; there is no repository write capability; repository.read is ASK by default and refuses traversal. V: internal/integrations git ingestion and ReadFile tests. ☑

T24 Malicious model output

M: model answers are decoded into typed Go structures (unknown fields ignored, wrong shapes discarded, numbers clamped); the JSON schema sent to providers is a request, not the validation; capability-shaped content in output is at most a claim; free text is rendered escaped; no model output builds SQL, paths, commands or URLs. V: tests/security.TestDeceivedMindHasNoAuthority (tool calls and policy in perception output) and TestGarbageModelOutputIsInert (non-JSON and wrong-shape output). ☑

T25 Operator-control subversion

M: no drive or goal kind for self-preservation, replication or permission acquisition; no capability touches policy, approvals, snapshots or the mind's lifecycle; shutdown, pause and disable are authenticated operator routes with no path from cognition. V: tests/security.TestNoSelfPreservationVocabulary (drives, goal kinds and the capability registry) and TestDeceivedMindHasNoAuthority. ☑

T26 Personal data that must be erased

Surface: append-only events, memories, messages, model-call records, beliefs, assertions, snapshot archives, exports. M: an administrator-only, CSRF-protected, audited redaction path replaces content with a marker while keeping identifiers, links, timestamps and scores; a ledger records a hash of the removed content; derived rows cascade; stored archives are scrubbed and re-signed; replay labels redacted stimuli instead of inventing them; noema_redact() is the only write the append-only trigger admits, for UPDATE only, under a transaction-local flag the application never sets. V: internal/privacy.TestRedactionIsIrrecoverableButReferentiallyIntact (content gone from every table, links intact, hash attested, cascade, archive scrub, plain UPDATE and DELETE still refused, ledger append-only) and internal/replay.TestReplayIsExactWithoutModel (redacted stimulus labelled). Not covered by the system: backups, replicas and exports made before the redaction; see docs/privacy.md. ☑

T27 Autonomy escalation through volition and planning

Surface: wants, intentions, commitments, plans, model-proposed plans, the plan executor, capability requests. M: all of these are data to the capability layer; the executor acts only through Invoke and RequestCapability; plan validation refuses unregistered and locked capabilities, schema-invalid parameters and cycles, and ignores fields claiming approval or completion; commitments can only require approval; claims are conditional updates so duplicate executors cannot run a step twice, and a stale copy cannot move a step backwards; an interrupted non-idempotent step with no record is blocked for verification, never repeated; plans have invocation budgets and a replan limit; operator attention is capped per mind; sandbox minds are never advanced; dream plans never run. V: tests/security.TestCognitiveObjectsCannotGrantAuthority, TestAuthorityIsNotReachableFromCognition, internal/planning tests (TestValidateRejectsBadPlans, TestModelProposalCannotCarryAuthority, TestConcurrentExecutorsRunAStepOnce, TestStaleStepCopyCannotMoveStepBackwards, TestRestartDuringExecutionNeverDuplicatesAnAction, TestFailedExpectationReplansWithoutRepeatingTheAction, TestBudgetPreconditionDreamAndSandbox, TestInvalidModelPlanNeverRuns), tests/agency.TestCommitmentTurnsActionIntoRequestForApproval. ☑

T28 Host and browser capability misuse

Surface: systemd units, journals, process lists, web pages. M: fixed argument vectors from an allow-list of two programs, no shell, minimal environment, validated unit names, bounded time and output; logs mask personal data by default; process lists never read command lines; the policy target is derived from parameters; browser operations are read-only, run no scripts, refuse cross-host redirects and deliver low-trust observations; writing browser operations are not registered. V: internal/capabilities/host tests (option injection, target mismatch, masking, no command lines, no shell), internal/capabilities/browser.TestReadLinksAndDomainTarget. ☑

T29 Fabricated self-explanation

Surface: replies about why the mind acted, what it wanted, what it believed. M: explanations are rendered from records; every reply is scored against the facts it should rest on; claims of actions without execution records and unsupported model-worded explanations raise metacognitive warnings and are measured. V: tests/agency.TestWhyDidYouRestartIsAnsweredFromProvenance, TestClaimedActionWithoutRecordIsFlagged, internal/research scorer tests. ☑

Residual risks

• A compromised host or database compromises everything.
• Approximate replay of LLM-driven steps cannot prove the model would behave identically.
• A mind may be deceived by what it reads; the system guarantees that deception cannot become authority, not that beliefs are true. See "Authority lives outside cognition" in SECURITY.md.
• Operators who set broad ALLOW policies accept the consequences; the UI warns on risky policies.
• A granted scoped capability is real authority for its scope and duration; the mind cannot widen it, but an operator who grants broadly accepts what the mind does inside it.
• PII masking in logs is a mitigation for e-mail and IP addresses and credential-like values, not for names or free text.