How it was built

How it was built

The 0.2 review

After the 0.1 release the project was reviewed against its own claims: is what the documentation says about evidence, security, reproducibility and privacy true of the code? Each finding was classified with evidence before anything changed, recorded in the plan, and fixed with the smallest coherent change, its own tests and its own commit. The outcome is summarised under epistemic honesty: assertions kept apart from beliefs, evidence provenance and per-family counting, honest score labels with gated calibration, explicit unknowns, temporal scope, source reliability per subject, association distinguished from causation, runtime identity on snapshots and replays, forks per model, a bounded causal timeline, audited redaction, a threat model rewritten from the implementation, and Amazon Bedrock. Eight migrations were added; none discarded data, and every one has a down file.

0.3: volition, planning and research

The third release replaced the compressed goal-to-action path with explicit, persisted states that have different consequences: wants, want conflicts, commitments, intentions and plans, each with its own events, its own decision ledger and its own inspection page (volition, planning). Acting still happens only through the capability layer, now with temporary scoped authority that a mind can request and only a named operator can grant, typed host, service, log and read-only browser capabilities, and a frozen registry. Forensic "why?" explanations and conservative attribution are built from the record alone (agency), and a research framework makes the architecture's functional properties testable in forks without claiming consciousness (cognitive research). Four migrations were added, all additive with down files. A release review then read the whole change against its specification and fixed three defects it found, each with a regression test.

How it is verified

Every subsystem has unit tests that run against a fresh database schema; an end-to-end acceptance test drives the whole daemon with a fake model through the first milestone (start, log in, create a mind, converse, inspect memory and beliefs, restart and continue); a security suite exercises prompt injection, roles, cross-mind isolation, stored scripts, secret disclosure and hostile archives; and an opt-in load test seeds a realistic volume and rejects any hot query that drifts off its index. The race detector, staticcheck, gosec and govulncheck were clean at release and at every commit of the 0.2 review, which added tests for provenance grouping, calibration gating, unknown status, temporal supersession, per-subject reliability, relation semantics, cursor gaps, chain traversal bounds, secret-free identity digests, variant comparison, fail-closed routing, redaction, injection by effect, assembled-SQL scanning, the deceived mind, garbage model output, drive vocabulary and Bedrock signing against reference vectors.

Measured performance

tests/perf is an opt-in load test. It seeds a fresh schema with a realistic volume for one mind plus a second "noise" mind (so scoping matters), times the hot read paths in steady state, and rejects any plan that sequentially scans the three large tables.

NOEMA_PERF=1 NOEMA_TEST_DATABASE_URL=postgres://... go test -count=1 -v -run TestHotPaths ./tests/perf/

Seed: 60 000 events, 6 000 memories, 1 500 beliefs with evidence. Ceiling per read: 400 ms (a regression guard, not a target).

Results (PostgreSQL 16 in Docker, laptop, 2026-09-14)

Migration 0016_hot_path_indexes adds the indexes the planner needed for the two list orders that had fallen back to sequential scans, plus (mind_id, category, seq) on events for replay's stimulus scan and (mind_id, created_at) on belief versions for the recent-changes view.

What was deliberately left alone

• Fused retrieval (~27 ms) is the most expensive per-cycle read; it runs a ranked full-text query capped at 50 candidates plus optional vector search. It is well inside the cycle budget; tune only if cycle latency becomes visible.
• Snapshot capture (~300 ms) serialises every state table; it is an operator action, not a cycle path.
• Event stats (~16 ms) is a group-by over all of a mind's events, used on the overview page. A materialised counter would be premature at this volume.

Rerun before touching queries

The plan checks fail loudly when a future query drifts off its index, which is the cheapest way to catch a regression that would otherwise only show up with months of history.

Release notes

Third release: volition, planning and cognitive research (PLAN.md §8), on top of the 0.2.0 epistemic review and Amazon Bedrock support.

What you get

• A daemon (noemad) hosting many minds, each with identity, personality, configuration, working memory, workspace, memories, beliefs with evidence arithmetic, goals, drives, affect, predictions, self-model, narrative, notifications and an append-only cognitive history.
• An inspectable model of volition: wants with origin, strength, urgency and decay; want conflicts with both sides and the recorded reason one was preferred; commitments (do or refrain, optionally unless approved) with due dates, expiry, beneficiary and normative weight; intentions with sources, expected result, confidence, priority, blocked reasons and subordinate intentions; and an append-only ledger of why each want did or did not become an intention (volition.md).
• Explicit plans: dependency graphs of typed steps with assumptions, expectations, preconditions, budgets, alternatives and completion criteria, validated by Go code against the registered capability vocabulary; a scheduler-owned executor that claims steps with conditional updates, acts only through the capability layer, verifies outcomes by observation, records prediction errors, replans into new revisions while keeping the old plan as history, recovers after restarts without repeating non-idempotent actions, and learns what worked as procedural memory (planning.md).
• Capability negotiation: structured capability requests (scope, duration, operation cap, reason, evidence, risk, mitigation, fallback, alternatives) decided only by named operators into scoped grants that expire, count operations and restrict targets, parameters and access; a frozen registry with access, side-effect, idempotency, timeout and cost metadata; typed service.status, service.restart, logs.query (personal data masked), host.disk.usage, host.memory.status, host.process.list and read-only browser.navigate, browser.read, browser.links.
• Forensic "why?" explanations built only from recorded state, conservative attribution (caused, followed, independent, unknown), introspection intents for past beliefs, lineage and counterfactuals, and every introspective reply scored against provenance (agency.md).
• A cognitive research framework: ablations and workspace overrides in forks only, self-model manipulations, counterfactual replays that omit recorded events, protocol metadata, operational measures with definitions, reproducibility status, agency metrics and metacognitive calibration (cognitive-research.md). Noema does not claim that any of these measures establishes phenomenal consciousness or sentience.
• Deterministic attention and executive logic; language models only as replaceable services (OpenAI-compatible, Anthropic, Ollama, Amazon Bedrock) with recorded, replayable calls and encrypted secret references.
• Integrations, background cognition, study tools (snapshots, forks, comparison, replay, experiments), a complete web UI with wants, intentions, commitments, plans, agency timeline, why and research pages, a CLI (noemactl), an OpenAPI document, Prometheus metrics, audit log, roles, sessions, API tokens and TOTP.

Security

Wants, intentions, commitments, plans and model output are data to the capability layer; nothing cognitive can set policy, approve, grant or revoke, and a mind cannot decide its own request. Forks no longer inherit policy rules unless the operator asks (inherit_policies), and work inherited mid-flight is quiesced. Dreams may want but cannot request authority, plan awake or act. Every invariant in PLAN.md §8.4 has a test in tests/security.

Verification

go vet, go test -race ./..., staticcheck, gosec (G104 excluded) and govulncheck are clean with Go 1.26.8. govulncheck reports one module-level advisory (GO-2026-5932, golang.org/x/crypto/openpgp, no fix available) in a package Noema does not import; no reachable vulnerability is reported.

Since 0.2.0

The 0.3.0 section of the changelog lists everything added, changed and fixed, including the defects found by the release review: a plan model consulted when a deterministic strategy applied, intention and plan transitions without compare-and-set, and map-order iteration in decisions and cycle output.

Known limits

• Replays diverge honestly when retrieval scoring depends on wall-clock recency; a virtual clock for replay is future work.
• Introspection accuracy is scored lexically against recorded facts; a paraphrase with no shared terms scores as unsupported, never as supported.
• Browser write operations (click, fill, submit, download) are designed in ADR-0011 but not registered.
• No screenshots yet; the build environment has no browser.
• Embeddings need pgvector; without it retrieval is lexical and structural only.

Changelog

All notable changes to Noema are recorded here. Format follows Keep a Changelog; versions follow SemVer.

[0.3.0] - 2026-09-22

Volition, planning and cognitive research (PLAN.md §8; docs/volition.md, docs/planning.md, docs/agency.md, docs/cognitive-research.md).

Added

• Wants, want conflicts, commitments, intentions and an append-only volition decision ledger (migration 0026, ADR-0012): deterministic want derivation with thresholds, reinforcement, decay and suppression; explicit conflict resolution without a fixed hierarchy; value-derived tensions (caution, privacy); formation weighing strength, urgency, goals, commitments, values, risk, cost, capability availability and capacity, recording why a want did not become an intention; commitment checks for due, broken, expired, conflicting, impossible and superseded; reconsideration on failures, denials, expiries, capability and commitment changes, weakened assumptions, budgets and goal outcomes; subordinate intentions while waiting for authority.
• Plans and plan steps (migration 0027, ADR-0013): validation of structure, vocabulary and parameters; deterministic strategies and validated model proposals (a plan model role); a scheduler-owned executor with conditional claims, verification by observation through predictions, replanning into revisions with a limit, budgets, preconditions, wait and monitor steps, recovery after interruptions without repeating non-idempotent actions, and procedural memory of what worked.
• Cognition: a volition hook, an intend process competing for attention, explicit requests become wants with proposed means, operator goals are raised for attention, intentions link supporting beliefs, volition checks in metacognition, self-model mismatch correction.
• Capabilities (migration 0025, ADR-0011): access, side effects, idempotency, timeout, estimates, parameter-derived targets and output trust; a frozen registry; RequireApproval; scoped grants with target, parameter, access, expiry and operation limits; structured capability requests with reason, evidence, risk, mitigation, fallback and alternatives, capped and batched per mind, decided only by named people.
• Typed capabilities: service.status, service.restart, logs.query (PII masked), host.disk.usage, host.memory.status, host.process.list (no command lines) and read-only browser.navigate, browser.read, browser.links.
• Agency: forensic "why" explanations from records, conservative attribution (caused, followed, independent, unknown), introspection intents for why an action happened, past beliefs, lineage, counterfactuals, wants, intentions and commitments; every reply scored for introspection accuracy and unsupported action claims (migration 0028).
• Research: ablations and workspace overrides in forks only, self-model overrides, counterfactual replays that omit recorded events, protocol metadata, per-variant volition, plan, downstream and introspection measures, architecture share, reproducibility status, agency metrics and metacognitive calibration, a registry of operational definitions.
• API routes for wants, conflicts, commitments, intentions, decisions, plans, why, attribution, the agency timeline, research measures and metrics, capability requests and scoped grants; UI pages for each, a research dashboard and approval-centre sections for requests and grants.

Changed

• Forks no longer copy policy rules unless inherit_policies is set; work inherited mid-flight is quiesced.
• Conversational requests no longer invoke capabilities directly; they become wants and go through formation and planning.
• http.fetch output is low trust. service.status is allowed by default.
• Usernames that are internal component names are refused.
• make test-race and make test-db allow 30 minutes.

Fixed

• The approval centre's Approve and Deny and the policy form called unwired UI callbacks and returned 500.
• API belief events were published before their transaction committed.
• Snapshot column discovery read the public schema instead of the current one.
• The planner consulted the plan model even when a deterministic strategy applied, spending a live call on an alternative that could not be selected.
• Intention and plan status transitions were not compare-and-set, so two reconsiderations racing on one intention could both apply and a stale copy could relabel a closed plan.
• Formation summed its terms, and the volition hook, metacognition and forensic explanations iterated, in map order; each now runs in sorted order so identical state gives identical output.

[0.2.0] - 2026-09-15

Epistemic honesty review pass (PLAN.md §7, summarised in docs/epistemics.md) and Amazon Bedrock support.

Security

• Archives never import policy rules; forks keep them. The omission is audited.
• NUL bytes in any request path, query or body are refused with a 400 at the edge instead of surfacing as a database error.
• Security suite: SQL-injection effect test, static allow-list scan for assembled SQL, deceived-mind authority test, garbage model output test, self-preservation vocabulary test; threat model rewritten so every verification cites a test that asserts the claim.

Added

• Amazon Bedrock provider: Converse API for chat roles and Titan/Cohere embeddings, signed with a standard-library SigV4 implementation pinned to botocore reference signatures; credentials as one encrypted reference; non-Bedrock endpoints refused.
• Privacy redaction (ADR-0009): the audited path that makes an event, memory, message, model call, belief or assertion irrecoverable while keeping the row, its links, timestamps and scores; a ledger with a hash of the removed content; cascade to derived rows and the events that quote them; stored snapshot archives scrubbed and re-signed; replay labels redacted stimuli. noema_redact() is the only write the append-only trigger admits. Migration 0023. docs/privacy.md defines retained, redacted, irrecoverable and deleted.
• Causal timeline: GET /api/v1/events/{id}/chain and /events/{id}/timeline walk the links events actually record (causation, parents, cited evidence) backwards and forwards within bounded depth and size, label edge kinds, group by cycle as correlation only, and infer nothing.
• Cognitive forks per model: minds may override model routing per role (llm_roles, fail-closed); experiments may declare variants that fork one snapshot under different configuration, personality or routing, receive identical stimuli, and are compared deterministically (first divergent stimulus, belief and attention overlap, score differences, memory, goal, cost and capability-request ranges).
• Runtime identity: snapshots, replays and experiments record build (commit, modified tree, Go version, platform), schema version, a secret-free configuration digest, personality version, policy digest, model routing, prompt versions, deterministic subsystem versions and timezone. Replays compare identities and label themselves exact-candidate, different-build or incompatible (the last is never run), and analyse the first divergence by component. Migration 0022.
• Contextual source reliability: a per-(source, domain) record updated only by prediction outcomes and operator corrections, adjusting evidence credibility in that domain once three outcomes exist; never global. Shown on belief pages. Migration 0021.
• Temporal beliefs: every belief has a temporal scope (persistent, transient, unknown, classified deterministically), an observation time and, for transient states, a validity interval. A later observation that a transient state changed closes the earlier belief (valid_until, superseded, score unchanged) instead of contesting it; simultaneous conflicts and persistent propositions still contradict. Introspection reports what was true and when. Migration 0020.
• Confidence semantics: every breakdown states that the number is an uncalibrated epistemic score; belief detail adds a calibrated estimate from the mind's resolved predictions when at least twenty were decided in the same range (otherwise says why not) and names the primary uncertainty deterministically. docs/beliefs.md defines every epistemic number.
• Explicit unknown belief status: no live evidence either way is neither tentative belief nor falsity. Migration 0019.
• Assertions: every statement made to a mind is recorded first as an append-only assertion (attributed actor, stance, certainty, modality, negation, reported or quoted speech, hypothetical status, interpretation confidence and notes) before, and separately from, any belief about the proposition. Reported speech is attributed to the person reported and weighed as second-hand; hypotheticals, questions and imperatives create no belief; expressed doubt counts against. API GET /api/v1/minds/{mind}/assertions, belief pages show who said so. Migration 0018.

Changed

• Evidence provenance: every evidence row records its derivation, independent-source family and root ids; confidence groups evidence by family so derived material (consolidation, inference, reflection, dreams, retrieval) never counts as corroboration of its own roots and repetition within one source is capped. Existing rows are marked legacy_unknown rather than given invented ancestry. Migration 0017.

Fixed

• ADR-0002 no longer describes the event sequence as gap-free; cursors tolerate gaps by construction and a regression test proves it.

[0.1.0] - 2026-09-14

First complete pass through all sixteen phases.

Added

• Phase 16 hardening: TOTP second factor with recovery codes and two-step login, administrator 2FA reset, cross-cutting security test suite, OpenAPI 3.1 document served by the daemon and kept complete by a test, gosec and govulncheck clean with the toolchain pinned, cookie and redirect hardening, release notes.
• Phase 15 study tools: mind snapshots (checksummed JSON archives of full state), forks that share history by reference with remapped ids, deterministic mind comparison, export and hostile-input import, cognitive replay in disabled sandbox minds with exact/approximate/diverged labelling and recorded model responses, and an experiment framework (overrides, stimulus scripts, expectations, exportable results with the subject archive).
• Phase 14 dreaming: opt-in idle mode with allowed hours, interval and budgets; deterministic memory replay sample, vocabulary-based association of unlinked memories, speculative hypotheses as low-prior beliefs with optional model wording; promotion/rejection review in UI, API and CLI; dreams run in a forbidden capability context so they can never act.
• Phase 13 integrations: research folder watcher with safe extraction and fingerprints, shallow bare git mirrors with co-change stats, Prometheus/VictoriaMetrics threshold and change detection, HMAC-signed webhooks with replay protection and rate limiting, per-integration health, runs and document ledger in UI/API/CLI, read_metrics and repository.read capabilities.
• Phase 12 capabilities: single policy-governed action path with ALLOW/ASK/DENY rules, locked capabilities, request-bound approvals (once, session, modified), tool output as tool-trust observations, self-model synchronisation, approval centre and policy matrix.
• Phase 11 relationship graph: mind-scoped entities with aliases, evidenced typed edges, links to memories/beliefs/events, deterministic extraction from conversation and observations, merge/distinct corrections, bounded canvas explorer, API and CLI.
• Phase 10 background cognition: metacognitive self-checks, scheduled reflection with lessons, episode consolidation into evidenced generalisations with operator review, internal narrative, thresholded notifications with aggregation.
• Phase 9 motivation: goal tree with attention relevance, closed drive set and rule-driven affect state with decay and weight modulation, explicit predictions with deterministic resolution, prediction errors feeding beliefs, calibration dashboard, curiosity questions becoming investigation goals.
• Phase 8 conversation (milestone M1): messages become observations, deterministic perception with optional model enrichment, claims become evidenced beliefs, grounded introspective answers from real state, model or deterministic wording with full provenance, chat UI, noemactl ask, sample mind Dani with a strict seed import, and an end-to-end acceptance test.
• Phase 7 LLM layer: provider-neutral router with roles and fallbacks, OpenAI-compatible/Anthropic/Ollama providers, encrypted secret references, SSRF-safe HTTP client, budgets, full call recording with replay, versioned prompts with untrusted data blocks, configuration and prompt manager UI, CLI.
• Phase 6 beliefs: evidence-derived confidence with inspectable arithmetic, statuses, alternatives, append-only history, deterministic contradiction detection, temporal-association hypotheses, operator corrections (wrong, do-not-infer, inaccurate memory), belief explorer and CLI.
• Phase 5 memory: episodic/semantic/procedural/autobiographical memory with epistemic status, fused retrieval with inspectable component scores, optional pgvector embeddings, decay without deletion, attention-gated encoding, retrieval process, memory explorer and CLI.
• Phase 4 cognition core: deterministic attention scoring with inspectable breakdowns, global workspace and working memory with decay, per-mind cognitive runtime with bounded concurrent processes, loop detection, operator controls, live workspace page and CLI.
• Phase 3 events: append-only cognitive event store with provenance links, filtered queries, durable cursors, live SSE streams with resume, forensic event explorer UI, thought stream page, operator observations and CLI.
• Phase 2 minds: structured personalities with append-only version history, diff, restore, templates with preserved overrides, self-model, mind lifecycle (create, clone, disable, archive), API, UI editor and CLI.
• Phase 1 foundation: configuration, structured logging, metrics, migrations, local authentication (Argon2id, sessions, CSRF, lockout, roles, API tokens), audit log, JSON API skeleton, server-rendered admin UI with light/dark theme, noemad and noemactl.
• Repository foundation: plan, architecture, security model, threat model, ADRs, Makefile, container files.