Sign in to edit tickets from this page.

← all tickets · home

Postgres-native world store: turns, audit events, attempts, registry, deletion, and execution provenance

resolved 293a300e-abf3-4f7c-85a4-f7129b742769

created_at
2026-04-26
updated_at
2026-04-26
priority
P1
resolved_at
2026-04-26
resolution
accepted

Body

Postgres-native world store: turns, audit events, attempts, registry, deletion, and execution provenance

P1, feature, multi-phase. Continues the trajectory begun in ticket 7d14ef0b (database-backed scenario store). Revised per consultant review.

Summary

Move the world execution layer — world metadata, per-turn state snapshots, audit events, attempt records, the worlds registry, and deletion semantics — into Postgres as a single clean cutover. After this ticket, no production code path reads or writes world state from the filesystem.

This is not a faithful file-to-table port and not an online dual-write migration. We have no users, no backward-compatibility obligation, and no legacy data we need to preserve. The substrate is replaced wholesale. Pre-deploy purges any existing worlds; post-deploy starts from an empty world layer with the new shape.

The migration encodes execution semantics as schema-level invariants. A turn is committed iff a single Postgres transaction successfully writes the attempt-row update, the new turn snapshot, every audit event for that turn, every event-entity row, and the world's current-turn pointer. There is no partial commit. There is no audit-write-after-snapshot. The transaction is the contract.

This ticket also fixes one execution-provenance gap the prior migration left open: audit events now record component hashes (cognition profile, perceive system, intend system, adjudicate system, adjudication schema) at execution time. Reverse-lookups like "every world that actually ran a turn through this adjudicate system" become single SQL queries instead of recomputed-from-snapshot walks.

Background

The scenario store is in Postgres. Cognition components, profiles, environments, entities, scenario manifests, names, and derivations are content-addressed and queryable. The seam runs through the world layer: world metadata lives in meta.json, turn snapshots live in turns/turn_NNNNNN.json, audit events live in audit/events.jsonl, attempts live in attempts.json, and the worlds registry is rebuilt at startup from a directory walk of /var/lib/chukwa/worlds/.

The split makes relational queries that span worlds and scenarios expensive or impossible. The placeholder world_count = 0 on every ScenarioSummary exists precisely because there is no way to JOIN today. The scenario_hash invariant shipped in 7d14ef0b (WorldMeta.scenario_hash == scenarios.hash) is the foundation; this ticket finally lets that join be exercised.

The destination, not addressed by this ticket but worth keeping in view, is automated cognition exploration: many worlds run in parallel, evaluations queried over their histories, mutations explored via a genetic-algorithm-style loop. Every architectural decision in this ticket is in service of that destination. The most important consequence is that this ticket cannot defer execution provenance: the eventual evaluation layer needs to ask "did this behavior happen under this exact prompt hash" and the audit log must answer it without recomputing from a snapshot.

Out of scope

Explicitly NOT included in this ticket. Each will be addressed separately or deferred.

  1. Ticketing system migration. The ticketing system remains on the filesystem under {data_root}/tickets/. No changes here.
  2. OAuth tokens, OAuth client config, session secret. These remain file-backed for now. Their eventual home is a separate concern.
  3. Documentation. No docs/terms.md, no docs/scenarios.md, no docs/operations.md, no module-doc or crate-doc prose updates. A fresh documentation ticket will be filed against the post-migration shape after this resolves.
  4. Rollback, fork-from-turn, time-travel. These lifecycle features become tractable once the substrate is in Postgres. They are not delivered by this ticket. A future ticket will design and ship them on top of the new substrate.
  5. UI. No new web routes for the new shape, no rendering changes beyond what is required to keep existing pages working against the new data source. A fresh UI ticket will be filed after this resolves.
  6. Evaluation layer, genetic-algorithm layer. Future work. This ticket only delivers the substrate they will be built on.
  7. touched_components schema upgrade. The current string-based encoding (cognition_profiles[subject].adjudicate_system) stays as-is. If the future evaluation layer needs structured queries over derivation diffs, that is a separate ticket.
  8. Multi-process workers and queued attempts. This ticket has no queued attempt state and no worker-pool design. The single-writer-per-world rule continues. Future multi-process safety is preserved by the lease-and-CAS pattern, but not exercised.

If a phase produces work that touches any of the above, that work is reverted or removed before the phase is declared done.

Migration philosophy

Clean cutover, not online migration

Pre-deploy purges all worlds via delete_world. Deploy applies migrations. Post-deploy verifies the world tables are empty. The first new world is created against the new substrate. No data carries forward. No file fallback exists in production. No "attach from disk" code path remains.

This is the right shape because:

Postgres is the source of truth for world execution

Not a cache. Not a write-through. Not a sync target. The database holds the canonical state for every world. The binary cannot start without a working Postgres connection and a successfully-applied migration. There is no in-memory or filesystem fallback in production. Memory-backed implementations may exist gated behind #[cfg(test)] or --features test-fixtures for unit tests; production builds must not be able to construct them.

Lifecycle semantics encoded as schema invariants

The schema enforces what counts as committed, what counts as failed, and what counts as in-flight. The commit_turn operation is one transaction; partial commits are unrepresentable. Restart recovery is a startup query that converts orphaned running attempts to interrupted. Deletion is a status transition, not a filesystem absence.

Implementation phases, single storage replacement

The work is sequenced into phases for reviewability and durable handoff between sessions. Each phase produces a commit on a feature branch. The cutover happens at deploy; until then, the new substrate is built alongside the old code without affecting production. After cutover, the old code paths are removed.

Architectural decisions

These are the load-bearing decisions, surfaced explicitly so the implementing handler does not relitigate them.

(D1) Postgres is source of truth for world execution. No production file fallback. Trait objects (Arc<dyn WorldStore>) are allowed for dependency injection and for tests, but runtime backend selection is not — the production binary constructs PostgresWorldStore unconditionally, with no environment-driven branching.

(D2) Turn commit is one transaction. Attempt update, turn snapshot insert, audit event inserts, event-entity inserts, world current-turn pointer update — all or none. If any part fails, the entire turn fails and the world's state is unchanged.

(D3) Audit consumers read from a durable cursor over world_audit_events. No live world-audit SSE consumers exist today. When such consumers are added later, they will read via cursor pagination over the audit table, with LISTEN/NOTIFY as a wake-up hint only — never as the data channel.

(D4) Component hashes are recorded at execution time on every audit event that depends on them. The kernel computes the relevant hashes when it builds the audit event input. Reverse-lookups do not recompute from world snapshots.

(D5) Drop scenario_snapshot from world state. The world is bound to a scenario by hash. The seeded state lives in world_turns(turn_number=0).state. The scenario manifest is recoverable via the scenario store. Storing a snapshot copy on the world is duplicative and risks drift.

(D6) Lease-based attempt claiming. When an attempt is started, the world's active_attempt_id is set in the same short transaction that creates the attempt with status='running'. The LLM cognition runs without any DB transaction held. Commit or fail is a separate short transaction. PostgreSQL advisory locks are not used as the primary in-flight representation; the lease column is durable and inspectable. A partial unique index enforces at most one running attempt per world.

(D7) world_turns.state stores the world's mutable state only. Environments, entities, simulation_time. NOT cognition profiles. NOT chronon_seconds. NOT the turn number itself; turn number is the row's primary-key component. Cognition profiles are immutable scenario content; they are loaded via worlds.scenario_hash → scenarios → cognition_profiles when the kernel needs them. Storing them in every turn snapshot is wasteful and risks drift. The schema discipline is enforced by a dedicated PersistedWorldState DTO — not by remembering to skip fields when serializing World.

(D8) Worlds, attempts, audit events, and turns are not content-addressed. They are temporal records with normal relational identity. state_hash on world_turns is an integrity check, not the row's identity. Identity is (world_slug, turn_number).

(D9) Deletion is durable status, not filesystem absence. A deleted world has status='deleted' and deleted_at set. Default list_worlds excludes deleted; an explicit flag returns them. There is no in-memory tombstone map. Deletion is rejected when a world is busy (has an active attempt). Hard deletion (purge) is a separate, explicit operation, not in scope here.

(D10) Restart recovery is automatic and explicit. On binary startup, before MCP/HTTP serving begins, any attempt with status='running' is transitioned to status='interrupted' with a recorded failure_reason, and any world with active_attempt_id pointing at one of those is cleared.

(D11) Attempts have no queued state. The MCP run_turn handler starts the attempt with status='running' directly, captures the world's lease, and spawns the cognition task — all in one short transaction. There is no queue table, no separate worker pool, and no queued → running transition. If a process crashes after starting an attempt but before commit, restart recovery transitions the attempt to interrupted. This keeps the lifecycle discipline minimal. Multi-process worker queues are a future concern; the lease pattern leaves room for them without requiring them now.

Schema

A new migration migrations/0002_world_store.sql adds the following. The order matters because of foreign keys: enums first, then worlds, then attempts, then the FK from worlds.active_attempt_id to attempts, then world_turns, then world_audit_events, then world_audit_event_entities.

Enums

CREATE TYPE world_status AS ENUM ('active', 'deleted');

CREATE TYPE attempt_status AS ENUM (
    'running',
    'committed',
    'failed',
    'interrupted'
);

Note: there is no queued state. See D11.

worlds

CREATE TABLE worlds (
    slug              label_text PRIMARY KEY,
    name              TEXT NOT NULL,
    scenario_hash     sha256_hex NOT NULL REFERENCES scenarios(hash),
    created_from_ref  JSONB NOT NULL,
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),

    status            world_status NOT NULL DEFAULT 'active',
    deleted_at        TIMESTAMPTZ,
    deleted_reason    TEXT,

    current_turn      BIGINT NOT NULL DEFAULT 0 CHECK (current_turn >= 0),
    active_attempt_id UUID,
    next_event_seq    BIGINT NOT NULL DEFAULT 1 CHECK (next_event_seq >= 1),

    CHECK ((status = 'deleted') = (deleted_at IS NOT NULL))
);

CREATE INDEX worlds_scenario_hash_idx ON worlds(scenario_hash);
CREATE INDEX worlds_status_created_idx ON worlds(status, created_at DESC);

Notes:

attempts

CREATE TABLE attempts (
    attempt_id        UUID PRIMARY KEY,
    world_slug        label_text NOT NULL REFERENCES worlds(slug),
    status            attempt_status NOT NULL,

    enqueued_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    started_at        TIMESTAMPTZ NOT NULL,
    ended_at          TIMESTAMPTZ,

    worker_id         TEXT NOT NULL,
    turn_before       BIGINT NOT NULL CHECK (turn_before >= 0),
    attempted_turn    BIGINT NOT NULL CHECK (attempted_turn >= 1),
    produced_turn     BIGINT,
    produced_turn_ref TEXT,

    progress          TEXT,
    failure_reason    TEXT,
    delta             JSONB,

    CONSTRAINT attempts_world_attempt_unique UNIQUE (world_slug, attempt_id),

    CHECK (attempted_turn = turn_before + 1),

    CHECK (
        (status = 'running' AND ended_at IS NULL)
        OR
        (status IN ('committed', 'failed', 'interrupted') AND ended_at IS NOT NULL)
    ),

    CHECK (
        (status = 'committed'
            AND produced_turn IS NOT NULL
            AND produced_turn = attempted_turn
            AND produced_turn_ref IS NOT NULL)
        OR
        (status <> 'committed'
            AND produced_turn IS NULL
            AND produced_turn_ref IS NULL)
    ),

    CHECK (
        (status IN ('failed', 'interrupted') AND failure_reason IS NOT NULL)
        OR
        (status NOT IN ('failed', 'interrupted'))
    )
);

CREATE INDEX attempts_world_enqueued_idx ON attempts(world_slug, enqueued_at DESC);
CREATE INDEX attempts_world_status_idx ON attempts(world_slug, status);

-- The partial unique index that enforces at most one running attempt per world.
CREATE UNIQUE INDEX attempts_one_running_per_world_idx
    ON attempts(world_slug)
    WHERE status = 'running';

After attempts exists, add the active-attempt FK:

ALTER TABLE worlds
    ADD CONSTRAINT worlds_active_attempt_fk
    FOREIGN KEY (slug, active_attempt_id)
    REFERENCES attempts(world_slug, attempt_id);

Notes:

world_turns

CREATE TABLE world_turns (
    world_slug      label_text NOT NULL REFERENCES worlds(slug),
    turn_number     BIGINT NOT NULL CHECK (turn_number >= 0),
    turn_ref        TEXT NOT NULL,
    simulation_time TIMESTAMPTZ NOT NULL,
    state           JSONB NOT NULL,
    state_hash      sha256_hex NOT NULL,
    entity_count    INT NOT NULL CHECK (entity_count >= 0),
    attempt_id      UUID,
    committed_at    TIMESTAMPTZ NOT NULL DEFAULT now(),

    PRIMARY KEY (world_slug, turn_number),
    UNIQUE (world_slug, turn_ref),

    CONSTRAINT world_turns_attempt_fk
        FOREIGN KEY (world_slug, attempt_id)
        REFERENCES attempts(world_slug, attempt_id),

    -- Turn 0 is the seed; no attempt produced it. All later turns must have one.
    CHECK (
        (turn_number = 0 AND attempt_id IS NULL)
        OR (turn_number > 0 AND attempt_id IS NOT NULL)
    )
);

CREATE INDEX world_turns_attempt_idx ON world_turns(attempt_id);
CREATE INDEX world_turns_committed_idx ON world_turns(committed_at DESC);

Notes:

world_audit_events and world_audit_event_entities

CREATE TABLE world_audit_events (
    event_id                 BIGSERIAL PRIMARY KEY,
    world_slug               label_text NOT NULL REFERENCES worlds(slug),
    world_event_seq          BIGINT NOT NULL CHECK (world_event_seq >= 1),

    turn_number              BIGINT CHECK (turn_number IS NULL OR turn_number >= 0),
    turn_ref                 TEXT,
    attempt_id               UUID,
    attempt_status           attempt_status,

    event_type               TEXT NOT NULL CHECK (event_type <> ''),
    occurred_at              TIMESTAMPTZ NOT NULL DEFAULT now(),
    simulation_time          TIMESTAMPTZ,

    entity_id                TEXT,

    profile_label            label_text,
    cognition_profile_hash   sha256_hex REFERENCES cognition_profiles(hash),
    perceive_system_hash     sha256_hex REFERENCES perceive_systems(hash),
    intend_system_hash       sha256_hex REFERENCES intend_systems(hash),
    adjudicate_system_hash   sha256_hex REFERENCES adjudicate_systems(hash),
    adjudication_schema_hash sha256_hex REFERENCES adjudication_schemas(hash),

    event                    JSONB NOT NULL,

    UNIQUE (world_slug, world_event_seq),
    UNIQUE (event_id, world_slug),

    CONSTRAINT world_audit_events_attempt_fk
        FOREIGN KEY (world_slug, attempt_id)
        REFERENCES attempts(world_slug, attempt_id)
);

CREATE INDEX world_audit_events_world_seq_idx ON world_audit_events(world_slug, world_event_seq);
CREATE INDEX world_audit_events_world_turn_idx ON world_audit_events(world_slug, turn_number);
CREATE INDEX world_audit_events_type_idx ON world_audit_events(event_type);
CREATE INDEX world_audit_events_attempt_idx ON world_audit_events(attempt_id);
CREATE INDEX world_audit_events_adjudicate_system_idx ON world_audit_events(adjudicate_system_hash);

CREATE TABLE world_audit_event_entities (
    event_id   BIGINT NOT NULL,
    world_slug label_text NOT NULL,
    entity_id  TEXT NOT NULL,
    role       TEXT NOT NULL CHECK (role IN ('subject', 'touched', 'mentioned')),

    PRIMARY KEY (event_id, entity_id, role),

    CONSTRAINT world_audit_event_entities_event_fk
        FOREIGN KEY (event_id, world_slug)
        REFERENCES world_audit_events(event_id, world_slug)
        ON DELETE CASCADE
);

CREATE INDEX world_audit_event_entities_world_entity_idx
    ON world_audit_event_entities(world_slug, entity_id, event_id);

Notes:

New Rust types

Errors

#[derive(Debug, thiserror::Error)]
pub enum WorldStoreError {
    #[error("world `{0}` not found")]
    NotFound(String),

    /// Used both for "operation refuses to proceed because target world is deleted"
    /// and for "delete_world called on an already-deleted world."
    #[error("world `{0}` is deleted")]
    Deleted(String),

    #[error("world `{0}` already exists")]
    AlreadyExists(String),

    #[error("world `{slug}` is busy: attempt `{attempt_id}` is in flight")]
    Busy { slug: String, attempt_id: String },

    #[error("attempt `{0}` not found")]
    AttemptNotFound(String),

    #[error("invalid attempt transition: cannot go from `{from:?}` to `{to:?}`")]
    InvalidAttemptTransition { from: AttemptStatus, to: AttemptStatus },

    #[error("turn {turn_number} not found for world `{slug}`")]
    TurnNotFound { slug: String, turn_number: u64 },

    #[error("scenario `{0}` not found")]
    ScenarioNotFound(String),

    #[error("commit lost the race: world `{slug}` current_turn was {expected}, found {actual}")]
    CommitRaceLost { slug: String, expected: u64, actual: u64 },

    #[error("commit rejected: lease check failed for attempt `{0}`")]
    LeaseInvalid(String),

    #[error("invalid input: {0}")]
    Invalid(String),

    #[error("database error: {0}")]
    Database(String),
}

impl From<sqlx::Error> for WorldStoreError { /* ... */ }

Do not control-flow on the string inside Database(String). Use typed variants for caller-visible cases.

Persisted state DTO

The dedicated DTO that enforces D7. The kernel's World is NOT serialized directly into world_turns.state; instead, PersistedWorldState::from_world(&World) extracts only the mutable parts.

#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct PersistedWorldState {
    pub simulation_time: DateTime<Utc>,
    pub environments: IndexMap<Label, String>,
    pub entities: IndexMap<String, Entity>,
}

impl PersistedWorldState {
    pub fn from_world(world: &World) -> Self { /* ... */ }
    pub fn into_world(self, slug: String, scenario: &Scenario, turn: u64) -> World { /* ... */ }
    pub fn state_hash(&self) -> String { /* sha256(canonical_json(self)) */ }
    pub fn entity_count(&self) -> usize { self.entities.len() }
}

The into_world reconstitution call takes a scenario reference to attach cognition_profiles and chronon_seconds from immutable scenario data.

Inputs and results

pub struct CreateWorldInput {
    pub slug: Slug,
    pub name: Option<String>,
    pub scenario_ref: ScenarioRef,
}

pub struct CreateWorldResult {
    pub slug: Slug,
    pub name: String,
    pub scenario_label: String,
    pub scenario_hash: String,
    pub created_at: DateTime<Utc>,
}

pub struct ListWorldsFilter {
    pub include_deleted: bool,
    pub scenario_hash: Option<String>,
}

pub struct WorldSummary {
    pub slug: Slug,
    pub name: String,
    pub scenario_hash: String,
    pub scenario_label: String,
    pub status: WorldStatus,
    pub current_turn: u64,
    pub created_at: DateTime<Utc>,
    pub last_activity: DateTime<Utc>,
    pub attempt_count: u64,
}

pub struct ClaimedAttempt {
    pub attempt_id: AttemptId,
    pub world_slug: Slug,
    pub world: World,
    pub turn_before: u64,
    pub attempted_turn: u64,
    pub scenario_hash: String,
}

pub struct TurnCommit {
    pub attempt_id: AttemptId,
    pub world_state: PersistedWorldState,
    pub events: Vec<AuditEventInput>,
    pub delta: TurnDelta,
}

// Note: world_slug, turn_before, produced_turn, turn_ref, entity_count, and
// state_hash are NOT supplied by TurnCommit. They are derived from the attempts
// row or computed inside commit_turn. This keeps the authoritative source of
// truth in the store transaction.

pub struct AttemptFailure {
    pub attempt_id: AttemptId,
    pub failure_reason: String,
    pub events: Vec<AuditEventInput>,
}

pub struct AuditEventInput {
    pub event_type: String,
    pub entity_id: Option<String>,
    pub simulation_time: Option<DateTime<Utc>>,

    pub profile_label: Option<Label>,
    pub cognition_profile_hash: Option<String>,
    pub perceive_system_hash: Option<String>,
    pub intend_system_hash: Option<String>,
    pub adjudicate_system_hash: Option<String>,
    pub adjudication_schema_hash: Option<String>,

    pub touched_entities: Vec<TouchedEntity>,

    pub event: Value,
}

// Note: attempt_id, attempt_status, turn_number, and turn_ref are stamped by
// commit_turn/fail_attempt from the locked attempts row. The kernel does not
// supply them per event.

pub struct TouchedEntity {
    pub entity_id: String,
    pub role: TouchedEntityRole,
}

pub enum TouchedEntityRole {
    Subject,
    Touched,
    Mentioned,
}

pub struct AuditCursor {
    pub world_event_seq_after: i64,
}

pub struct AuditPage {
    pub events: Vec<AuditEvent>,
    pub next_cursor: Option<AuditCursor>,
}

pub struct AttemptStatusRecord {
    pub attempt_id: AttemptId,
    pub world_slug: Slug,
    pub status: AttemptStatus,
    pub enqueued_at: DateTime<Utc>,
    pub started_at: DateTime<Utc>,
    pub ended_at: Option<DateTime<Utc>>,
    pub worker_id: String,
    pub turn_before: u64,
    pub attempted_turn: u64,
    pub produced_turn: Option<u64>,
    pub produced_turn_ref: Option<String>,
    pub progress: Option<String>,
    pub failure_reason: Option<String>,
    pub delta: Option<TurnDelta>,
}

pub struct DeletedWorldSummary {
    pub slug: Slug,
    pub name: String,
    pub scenario_hash: String,
    pub scenario_label: String,
    pub created_at: DateTime<Utc>,
    pub deleted_at: DateTime<Utc>,
    pub deleted_reason: Option<String>,
}

AttemptId wraps Uuid; use the existing newtype pattern from the scenario store. DeletedWorldSummary replaces the prior in-memory DeletedWorldRecord shape; it is a query result, not a cached map entry.

WorldStore trait

#[async_trait]
pub trait WorldStore: Send + Sync {
    // World lifecycle
    async fn create_world(
        &self,
        input: CreateWorldInput,
    ) -> Result<CreateWorldResult, WorldStoreError>;

    async fn list_worlds(
        &self,
        filter: ListWorldsFilter,
    ) -> Result<Vec<WorldSummary>, WorldStoreError>;

    /// Returns active worlds. To fetch a deleted world, use
    /// `get_world_including_deleted`.
    async fn get_world(
        &self,
        slug: &Slug,
    ) -> Result<WorldDetails, WorldStoreError>;

    /// Returns a world regardless of status. Used by audit/forensic tooling.
    async fn get_world_including_deleted(
        &self,
        slug: &Slug,
    ) -> Result<WorldDetails, WorldStoreError>;

    /// Marks the world as deleted. Rejected with `WorldStoreError::Busy` if the
    /// world has an `active_attempt_id`. Rejected with `WorldStoreError::Deleted`
    /// if the world is already deleted. Rejected with `WorldStoreError::NotFound`
    /// if the slug does not exist.
    async fn delete_world(
        &self,
        slug: &Slug,
        reason: Option<String>,
    ) -> Result<DeletedWorldSummary, WorldStoreError>;

    // Attempt lifecycle
    /// Atomically creates an attempt with `status='running'` and acquires the
    /// world's lease (`active_attempt_id`). Returns the world state needed for
    /// cognition. Rejected with `WorldStoreError::Busy` if the world already has
    /// an active attempt.
    async fn start_attempt(
        &self,
        slug: &Slug,
        worker_id: &str,
    ) -> Result<ClaimedAttempt, WorldStoreError>;

    /// Commits a turn. Loads the attempt row inside the transaction and verifies
    /// all lease/status/turn-number conditions. See "Turn execution flow" for
    /// the verification list.
    async fn commit_turn(
        &self,
        commit: TurnCommit,
    ) -> Result<(), WorldStoreError>;

    /// Records the failure. Loads the attempt row inside the transaction and
    /// verifies the same lease/status conditions as commit_turn, except for
    /// produced_turn/current_turn advancement.
    async fn fail_attempt(
        &self,
        failure: AttemptFailure,
    ) -> Result<(), WorldStoreError>;

    /// Startup recovery. Transitions all `running` attempts to `interrupted` and
    /// clears the corresponding world leases. Returns the count of reconciled
    /// attempts.
    async fn reconcile_running_attempts(
        &self,
    ) -> Result<usize, WorldStoreError>;

    async fn get_attempt_status(
        &self,
        attempt_id: AttemptId,
    ) -> Result<AttemptStatusRecord, WorldStoreError>;

    async fn list_attempts(
        &self,
        slug: &Slug,
    ) -> Result<Vec<AttemptStatusRecord>, WorldStoreError>;

    // Turn reads
    async fn get_turn(
        &self,
        slug: &Slug,
        turn_number: u64,
    ) -> Result<Turn, WorldStoreError>;

    async fn list_turns(
        &self,
        slug: &Slug,
        from_turn: Option<u64>,
        to_turn: Option<u64>,
        limit: usize,
    ) -> Result<Vec<TurnSummary>, WorldStoreError>;

    async fn diff_turns(
        &self,
        slug: &Slug,
        from_turn: u64,
        to_turn: u64,
    ) -> Result<TurnDiff, WorldStoreError>;

    /// Returns the latest turn at-or-before `simulation_time`. Tie-break:
    /// `ORDER BY simulation_time DESC, turn_number DESC LIMIT 1`.
    async fn get_state_at(
        &self,
        slug: &Slug,
        simulation_time: DateTime<Utc>,
    ) -> Result<World, WorldStoreError>;

    // Audit reads
    async fn read_audit_events(
        &self,
        slug: &Slug,
        cursor: AuditCursor,
        limit: usize,
        filter: AuditFilter,
    ) -> Result<AuditPage, WorldStoreError>;

    async fn entity_history(
        &self,
        slug: &Slug,
        entity_id: &str,
        cursor: Option<AuditCursor>,
        limit: usize,
    ) -> Result<AuditPage, WorldStoreError>;
}

AuditFilter carries the same filters the existing EventQuery supports: event_type, entity_id, turn range, include_failed. It is translated to SQL predicates, not in-memory filtering. The include_failed flag controls whether events from failed/interrupted attempts are returned; default is false. See "Failed-attempt audit semantics" below.

Lifecycle invariants

These are the contract. Tests must verify each.

(L1) An active world is well-formed iff a worlds row exists with status='active' AND a corresponding world_turns row at turn_number=0 exists. Both are written in create_world's single transaction. Deleted worlds may retain their rows and turn history, but default world reads exclude them.

(L2) A turn N where N ≥ 1 is committed iff:

All four conditions hold or none do. Partial commit is unrepresentable.

(L3) A failed or interrupted attempt does NOT advance worlds.current_turn. Failed attempts write failure-related audit events, including attempt_failed. Interrupted attempts from restart recovery need not have audit events because the in-memory event buffer is gone. In both cases, worlds.current_turn remains unchanged from before the attempt began.

(L4) An interrupted attempt is the result of a process crash mid-turn. On startup, reconcile_running_attempts finds every attempt with status='running', transitions each to status='interrupted' with ended_at=now() and failure_reason='process restart before commit', and clears worlds.active_attempt_id for any world that pointed at one of them.

(L5) A world has at most one running attempt at a time. Enforced at the schema level by attempts_one_running_per_world_idx ON attempts(world_slug) WHERE status='running'. Also enforced at the trait level: start_attempt fails with WorldStoreError::Busy if worlds.active_attempt_id IS NOT NULL.

(L6) A deleted world has status='deleted', deleted_at set, and active_attempt_id=NULL. Default list_worlds excludes deleted worlds. get_world on a deleted world returns WorldStoreError::Deleted; get_world_including_deleted returns the world regardless. Deletion is rejected with WorldStoreError::Busy when the world has an active attempt; the caller must wait for the attempt to commit, fail, or be reconciled.

(L7) next_event_seq is per-world monotonic. Audit event rows for a world have strictly increasing world_event_seq values. Concurrent transactions touching the same world serialize through the row-level lock acquired on worlds during commit/fail. Sequence allocation happens in batch.

(L8) Commit and fail verify the lease at execution time. Both commit_turn and fail_attempt load the attempt row and the world row inside the transaction with FOR UPDATE, then verify:

If any check fails, the transaction aborts with the appropriate error variant. worlds.status='deleted' during commit/fail should be unreachable because deletion rejects busy worlds; if it occurs, fail loudly.

World creation flow

WorldStore::create_world is a single transaction for world state. Scenario resolution may create immutable scenario-store rows first; that is acceptable because scenario content is content-addressed and an unused scenario row is harmless. The world itself is not partially created.

0. Resolve scenario_ref
     → if name: resolve scenario name to scenario_hash
     → if hash: verify scenarios(hash) exists
     → if data: call the existing single scenario assembly path, store/resolve it,
       and return scenario_hash
     → if resolution fails, return ScenarioNotFound/Invalid before writing world rows

1. create_world(input) [TRANSACTION T0, short]
     → INSERT INTO worlds (
         slug, name, scenario_hash, created_from_ref,
         status='active', current_turn=0, active_attempt_id=NULL,
         next_event_seq=1
       )
     → Build initial World from scenario
     → state = PersistedWorldState::from_world(&seed_world)
     → state_hash = state.state_hash()
     → INSERT INTO world_turns (
         world_slug=slug,
         turn_number=0,
         turn_ref='turn_000000',
         simulation_time=state.simulation_time,
         state,
         state_hash,
         entity_count=state.entity_count(),
         attempt_id=NULL
       )
     → no seed audit events in this ticket
   [COMMIT T0]

If any insert fails, no world row and no turn-0 row remain.

World creation: created_from_ref normalization

created_from_ref stores provenance, not scenario content. It must never store a full inline scenario payload. Shapes:

{ "kind": "name", "input": "vending-leak-fix", "resolved_hash": "<sha256>" }

{ "kind": "hash", "input": "<sha256>", "resolved_hash": "<sha256>" }

{ "kind": "inline_data", "resolved_hash": "<sha256>" }

For kind='inline_data', the original data may already live in the scenario store as content-addressed components and manifests. Do not duplicate it in the world row.

Turn execution flow

The kernel's Runtime::run_turn is rewritten to follow this sequence. No DB transaction is held while LLM cognition runs.

1. start_attempt(world_slug, worker_id)
     [TRANSACTION T1, short]
     → SELECT worlds WHERE slug=? FOR UPDATE
     → verify status='active' AND active_attempt_id IS NULL
       (else: WorldStoreError::Busy or Deleted or NotFound)
     → SELECT world_turns WHERE world_slug=? AND turn_number=worlds.current_turn
     → load scenario by worlds.scenario_hash so the World can be reconstituted
       before the lease is acquired; if scenario lookup fails, abort before insert
     → INSERT INTO attempts (
         attempt_id, world_slug, status='running',
         enqueued_at=now(), started_at=now(), worker_id=?,
         turn_before=worlds.current_turn,
         attempted_turn=worlds.current_turn+1,
         progress='running cognition, adjudication, and commit'
       )
       (the partial unique index throws a duplicate-key error if a concurrent
        start_attempt sneaks in; that is the schema-level safety net for L5)
     → UPDATE worlds SET active_attempt_id=attempt_id WHERE slug=?
     [COMMIT T1]
     → returns ClaimedAttempt{ world: World reconstituted from PersistedWorldState
       + scenario lookup, attempt_id, ... }
     → MCP run_turn handler returns immediately to caller with attempt_id and
       status='running'

2. Run cognition (NO DB TRANSACTION HELD)
     → For each agent in turn order:
         → perceive(agent, world) → calls LLM
         → emit perception_emitted event into in-memory event buffer with
           execution-time component hashes attached
         → intend(agent, world, perception) → calls LLM
         → emit intent_formed event with execution-time hashes
         → adjudicate(agent, intent, world) → calls LLM with retries
         → emit intent_adjudicated event (or adjudication_rejected on retry)
           with execution-time hashes
         → if adjudicated successfully: apply mutation to working World copy

3. Build TurnCommit OR AttemptFailure
     → if all agents adjudicated successfully:
         → state = PersistedWorldState::from_world(&world_after)
         → events = [perception_emitted, intent_formed, intent_adjudicated, ...,
                    turn_complete]
       else:
         → AttemptFailure { events: emitted-so-far + attempt_failed,
                            failure_reason: ... }

4a. commit_turn(commit) [TRANSACTION T2, short]
     → SELECT * FROM attempts WHERE attempt_id=? FOR UPDATE
     → SELECT * FROM worlds WHERE slug=attempt.world_slug FOR UPDATE
     → Verify (else: appropriate error variant):
         attempt.status = 'running'
         worlds.status = 'active'
         worlds.active_attempt_id = attempt_id
         worlds.current_turn = attempt.turn_before
     → produced_turn = attempt.attempted_turn
     → turn_ref = format!("turn_{:06}", produced_turn)
     → state_hash = commit.world_state.state_hash()
     → entity_count = commit.world_state.entity_count()
     → Allocate event sequence numbers in one update:
         UPDATE worlds
         SET next_event_seq = next_event_seq + $event_count
         WHERE slug = $slug
         RETURNING next_event_seq - $event_count AS first_event_seq
       Assign world_event_seq values: first_event_seq, first_event_seq+1, ...
     → UPDATE attempts SET
         status='committed', ended_at=now(),
         produced_turn=attempt.attempted_turn,
         produced_turn_ref=turn_ref,
         delta=?
     → INSERT INTO world_turns (
         world_slug, turn_number=attempt.attempted_turn,
         turn_ref, simulation_time=commit.world_state.simulation_time,
         state=commit.world_state, state_hash,
         entity_count, attempt_id, committed_at=now()
       )
     → For each event in commit.events:
         → INSERT INTO world_audit_events with:
             world_slug=attempt.world_slug,
             world_event_seq=allocated seq,
             turn_number=attempt.attempted_turn,
             turn_ref=turn_ref,
             attempt_id=attempt.attempt_id,
             attempt_status='committed',
             event_type / entity_id / simulation_time / component hashes / event
         → For each touched_entity: INSERT INTO world_audit_event_entities
     → UPDATE worlds SET current_turn=attempt.attempted_turn,
                         active_attempt_id=NULL
       WHERE slug=?
     [COMMIT T2]

4b. fail_attempt(failure) [TRANSACTION T2', short]
     → SELECT * FROM attempts WHERE attempt_id=? FOR UPDATE
     → SELECT * FROM worlds WHERE slug=attempt.world_slug FOR UPDATE
     → Verify (else: appropriate error):
         attempt.status = 'running'
         worlds.status = 'active'
         worlds.active_attempt_id = attempt_id
     → Ensure failure.events includes an attempt_failed terminator; if not,
       either append a standard one or reject with WorldStoreError::Invalid
     → Allocate event sequence numbers (same as commit_turn)
     → UPDATE attempts SET
         status='failed', ended_at=now(),
         failure_reason=?
     → For each event:
         → INSERT INTO world_audit_events with:
             turn_number=attempt.attempted_turn,
             turn_ref=format!("turn_{:06}", attempt.attempted_turn),
             attempt_id=attempt.attempt_id,
             attempt_status='failed'
         → INSERT INTO world_audit_event_entities
     → UPDATE worlds SET active_attempt_id=NULL WHERE slug=?
     [COMMIT T2']

The two transactions T1 (start) and T2 (commit/fail) are short. The expensive cognition step happens between them with no DB locks held. PostgreSQL row-level locks via FOR UPDATE are held only for the duration of T1 and T2, which is essentially I/O time.

The lease verification inside T2 is load-bearing: do not trust TurnCommit for turn_before; load the attempt row inside the transaction and use attempts.turn_before / attempts.attempted_turn as the source of truth. If a commit_turn ever fires LeaseInvalid or CommitRaceLost, that indicates a real bug — the single-writer rule should make it unreachable.

World deletion races

If a world is deleted while an attempt is in flight, two races are possible:

  1. delete_world arrives between start_attempt and commit_turn. Resolution: delete_world rejects with Busy because active_attempt_id IS NOT NULL. The attempt continues; its commit succeeds or fails.
  2. delete_world arrives during cognition, while no DB transaction is held, and the attempt's commit_turn arrives after. Resolution: with the busy-rejection rule, this is impossible — the deletion is blocked until the attempt clears.

Therefore: delete_world is rejected when active_attempt_id IS NOT NULL. The caller waits, retries, or uses a future explicit interruption/cancellation feature. Interruption/cancellation is not in scope here.

For fail_attempt, the deleted-during-attempt case cannot arise either. If fail_attempt observes worlds.status='deleted', treat it as a programming error and fail loudly.

Component hash provenance

Audit events that depend on cognition components MUST carry the relevant hashes at execution time. The kernel computes them as it builds the audit event input. The matrix:

event_typeprofile_labelcognition_profile_hashperceive_system_hashintend_system_hashadjudicate_system_hashadjudication_schema_hash
perception_emittedyesyesyes———
intent_formedyesyes—yes——
intent_adjudicatedyesyes——yesyes
adjudication_rejectedyesyes——yesyes
attempt_failed——————
turn_complete——————

Computation: when the kernel resolves which profile to use for an agent via the agent's cognition_profile label and the world's cognition_profiles map, it has the full CognitionProfile value in hand. Computing each component hash uses the canonical-json hashers already in src/canonical_json.rs. Cache the four sub-component hashes once per agent per turn; reuse for all events that turn touches.

The cognition profile hash itself is canonical_cognition_profile_hash(&CognitionProfile).

These hashes are required even when the world's cognition profiles all use components already stored in the scenario store. The point is execution provenance: at this exact moment, this exact prompt content was used. We assert that retroactively from the audit log, not by walking the world's snapshot back.

Failed-attempt audit semantics

For attempts with status = 'failed', audit events emitted during the attempt are written with turn_number = attempts.attempted_turn. This is the attempted turn, not a committed world_turns row — there is no row at that turn number for the world.

For attempts with status = 'interrupted', restart recovery usually cannot write attempt-local audit events because the in-memory event buffer is gone. The durable signal is the attempts row itself: status='interrupted', ended_at, and failure_reason='process restart before commit'.

Callers reading world_audit_events must be aware:

This makes include_failed a meaningful filter and avoids confusion when an attempted turn appears in audit events but has no corresponding world_turns row.

Seed audit events

Seed audit events at turn 0 (world creation) are OPTIONAL in the schema. create_world MAY emit one or more events in a future ticket, e.g. a world_created event, in which case those events consume world_event_seq values starting from 1.

For this ticket: do NOT emit seed events. The first event is perception_emitted from turn 1's first agent. This keeps create_world simple and matches current behavior. If no seed events are emitted, next_event_seq remains at 1 and the first audit event from the first run-turn takes seq 1.

If a future ticket wants world_created audit events for traceability, it adds emission to create_world; the schema already supports it.

Allocating audit sequence numbers

worlds.next_event_seq is allocated in batches, not one event at a time.

For a commit or failure with event_count > 0:

UPDATE worlds
SET next_event_seq = next_event_seq + $event_count
WHERE slug = $slug
RETURNING next_event_seq - $event_count AS first_event_seq;

Then assign:

first_event_seq
first_event_seq + 1
first_event_seq + 2
...

This avoids repeated row updates inside the commit/fail transaction. Because commit_turn and fail_attempt already lock the worlds row with FOR UPDATE, per-world event sequence allocation is serialized.

Audit cursor consumer model

The store exposes:

async fn read_audit_events(
    &self,
    slug: &Slug,
    cursor: AuditCursor,
    limit: usize,
    filter: AuditFilter,
) -> Result<AuditPage, WorldStoreError>;

Implementation shape:

SELECT e.*
FROM world_audit_events e
WHERE e.world_slug = $1
  AND e.world_event_seq > $cursor.world_event_seq_after
  -- default: exclude failed/interrupted attempt events
  AND ($include_failed OR e.attempt_status IS NULL OR e.attempt_status = 'committed')
  -- optional predicates: event_type, turn range
ORDER BY e.world_event_seq
LIMIT $limit;

If filtering by entity, use the side table so both primary subjects and touched/mentioned entities are included:

SELECT e.*
FROM world_audit_event_entities ee
JOIN world_audit_events e
  ON e.event_id = ee.event_id
WHERE ee.world_slug = $1
  AND ee.entity_id = $entity_id
  AND e.world_event_seq > $cursor.world_event_seq_after
ORDER BY e.world_event_seq
LIMIT $limit;

AuditPage.next_cursor is set when events.len() == limit, with world_event_seq_after equal to the last returned event's world_event_seq. Otherwise it is NULL.

Live consumers, when added later, will: open a connection, optionally LISTEN chukwa_world_events, then enter a loop:

  1. Issue read_audit_events with the current cursor.
  2. If results exist, deliver them to the consumer and advance the cursor.
  3. If empty, wait on NOTIFY or sleep with a timeout, then loop.

The NOTIFY wakeup is an optimization, not the data channel. A polling-only consumer is correct, just less responsive.

This ticket does NOT add LISTEN/NOTIFY. There are no live world-audit consumers today. The cursor read API is the only requirement. NOTIFY can be added in a follow-up ticket when a consumer needs it.

Restart recovery

WorldStore::reconcile_running_attempts is called from bin/chukwa-serve.rs AFTER migrations have applied and BEFORE any HTTP/MCP listener accepts traffic. Implementation:

BEGIN;

WITH interrupted AS (
    UPDATE attempts
    SET status = 'interrupted',
        ended_at = now(),
        failure_reason = 'process restart before commit'
    WHERE status = 'running'
    RETURNING attempt_id, world_slug
)
UPDATE worlds w
SET active_attempt_id = NULL
FROM interrupted i
WHERE w.slug = i.world_slug
  AND w.active_attempt_id = i.attempt_id;

COMMIT;

Returns the count of reconciled attempts. The startup code logs the count.

This is safe to run on every startup. On a cleanly-shut-down system, zero attempts will be in running and the operation is a no-op.

World deletion

WorldStore::delete_world(slug, reason) is a status transition. It must reject busy worlds. It does NOT clear active_attempt_id as a way to force deletion.

Implementation shape:

BEGIN;

SELECT slug, name, scenario_hash, created_at, status, active_attempt_id
FROM worlds
WHERE slug = $slug
FOR UPDATE;

-- If no row: NotFound.
-- If status='deleted': Deleted.
-- If active_attempt_id IS NOT NULL: Busy.

UPDATE worlds
SET status = 'deleted',
    deleted_at = now(),
    deleted_reason = $reason
WHERE slug = $slug
  AND status = 'active'
  AND active_attempt_id IS NULL
RETURNING slug, name, scenario_hash, created_at, deleted_at, deleted_reason;

COMMIT;

Returns DeletedWorldSummary. Errors:

Default list_worlds filters WHERE status = 'active'. The MCP list_worlds tool preserves the existing include_recently_deleted argument for compatibility with current behavior; the new implementation maps it to durable include_deleted=true rather than consulting in-memory tombstones.

Hard deletion (purge) is NOT in scope for this ticket. If we ever need it, it is a separate operation with an explicit cascade/ordering design.

Scenario world_count correctness

The scenario store's StoredScenario.world_count and ScenarioSummary.world_count placeholders, currently always 0, are now populated from active worlds:

SELECT s.hash, COUNT(w.slug) AS world_count
FROM scenarios s
LEFT JOIN worlds w ON w.scenario_hash = s.hash AND w.status = 'active'
GROUP BY s.hash;

Update the scenario store's queries to compute this. The decision is world_count = active worlds, not world_count = ever-existed worlds. Deleted worlds are excluded.

Tests must cover worlds created from:

This catches the same class of cross-layer hash-join bug that 7d14ef0b exposed.

MCP surface changes

Tools that change shape or implementation:

ToolChange
create_worldImplementation moves to WorldStore::create_world. Wire shape unchanged. The two-step "create then write back scenario_ref" pattern is gone; world creation is one transaction for world state.
list_worldsImplementation queries worlds table. include_recently_deleted flag preserved and mapped to durable deleted rows. last_activity is computed from MAX(committed_at) over world_turns for that world.
get_worldImplementation queries worlds + world_turns(turn=current). WorldDetails carries scenario_hash, current_turn, and the same fields surfaced today. Deleted worlds return WorldStoreError::Deleted through the default path.
delete_worldStatus transition, not directory rm. Returns DeletedWorldSummary mapped to the current MCP response shape. Busy worlds are rejected.
run_turnImplementation: start_attempt, return attempt_id immediately with status running. Background task, using the existing tokio task pattern, runs cognition and calls commit_turn or fail_attempt. No queued state.
get_turn_statusImplementation queries attempts table. Same response shape where possible.
list_attemptsQueries attempts table. Same response shape where possible.
get_turnQueries world_turns. Same response shape where possible.
list_turnsQueries world_turns. Cursor/range pagination via from_turn/to_turn + limit.
diff_turnsComputes diff from two world_turns.state JSONB values plus the audit events between them.
get_state_atQueries world_turns joined with worlds.scenario_hash to find the latest turn at-or-before simulation_time; tie-break by turn_number DESC. Reconstitutes the World.
get_eventsQueries world_audit_events with filter predicates as SQL. Cursor-based pagination via world_event_seq. The existing since parameter maps to cursor; optionally also expose a structured cursor argument if the MCP schema supports it cleanly.
entity_historyQueries world_audit_event_entities joined with world_audit_events.

No new MCP tool is added by this ticket. Post-migration LISTEN/NOTIFY support, if needed, gets its own follow-up ticket.

Web routes

The world detail page (/worlds/:slug) and the turn detail page (/turns/:slug/:turn_ref) currently read from WorldMeta::read and the on-disk turn snapshot files. After migration, they read from the WorldStore trait. Output HTML shape unchanged.

The world list page reads from list_worlds. Same.

No new routes. No rendering changes beyond what is required to swap the data source. The UI work for the new shape (linking, reverse lookups, derivation graph navigation) is a separate ticket.

Removed and deprecated symbols

Cleanup grep guards. All of these MUST return zero matches in production src/ code after Phase F:

Symbol / patternWhat it was
pub fn load_all in worlds.rsDirectory-walk world registry rebuild
WorldMeta::readmeta.json reader
WorldMeta::write_backmeta.json writer
pub scenario_snapshotThe redundant snapshot field on WorldMeta
struct DeletedWorldRecordIn-memory tombstone result/cache shape; replace with DeletedWorldSummary
mod persistenceThe src/persistence.rs module
mod turn_jobThe Jobs::save_locked / attempts.json file-writing path
audit/events.jsonlThe on-disk audit log
turns/turn_The on-disk turn snapshots
attempts.jsonThe on-disk attempts file
/var/lib/chukwa/worlds/The world directory path in code
meta.jsonWorld metadata file path

Some old structs may be reshaped rather than fully deleted if their names are still useful as in-memory API response types. The constraint is: no production code path reads or writes from /var/lib/chukwa/worlds/ after cleanup.

Test plan

Unit tests (#[cfg(test)] and --features test-fixtures)

Tests for input validation, error mapping, canonical state hashing, PersistedWorldState extraction/reconstitution, and the MemoryWorldStore if it exists. No live DB.

Postgres tests (--features postgres-tests)

Each test runs against a fresh schema (DROP + CREATE + migrate). Use RUST_TEST_THREADS=1 if tests share one local Postgres. Tests cover:

Target: approximately 45 new postgres tests.

Integration tests

tests/world_store.rs — end-to-end through WorldStore trait against a live Postgres. Includes restart-recovery test by seeding a running attempt, calling reconcile_running_attempts, and observing the world recoverable.

Live smoke (Phase H)

End-to-end against deployed pod. See "Smoke plan" section.

Phase plan

Phase A — Schema + foundation (additive, safe to deploy)

Acceptance: container build clean; cargo test --lib --features test-fixtures baseline plus any new module-level tests; migration-runner test covers 0002 forward and runner idempotency.

Phase B — WorldStore trait + Postgres implementation

Acceptance: cargo test --features test-fixtures,postgres-tests: all existing tests plus new Postgres tests, all green.

Phase C — Kernel rewrite

Acceptance: lib tests green; postgres tests include kernel-integration coverage; existing scenario/phase smoke tests either pass through to the new store or are rewritten to use it.

Phase D — MCP surface migration

Acceptance: all MCP-dispatcher tests pass; new tests cover cursor pagination, include_deleted path, busy delete, and failed-attempt audit filtering.

Phase E — Web surface migration

Acceptance: lib tests green; existing web-rendering tests pass against the new data source.

Phase F — Cleanup

Acceptance: every grep guard from "Cleanup grep guards" returns zero matches; full test suite green; container build clean.

Phase G — Pre-deploy purge + DB-pod state

Acceptance: list_worlds count = 0 against the pre-deploy binary; the new world tables in Postgres are empty.

Phase H — Deploy + live smoke

Acceptance: smoke green.

Phase I — Wrap-up

Acceptance: caller accepts.

Acceptance criteria

For ticket-level resolution.

  1. No production file-backed world state remains. No code path in the production binary reads or writes:

    • meta.json
    • turns/turn_NNNNNN.json
    • audit/events.jsonl
    • attempts.json
    • the /var/lib/chukwa/worlds/ directory

    Verified by grep guards.

  2. Startup requires Postgres for world state. Missing DATABASE_URL is fatal. No in-memory or filesystem fallback in production. bin/chukwa-serve.rs cannot be built such that WorldStore resolves to anything other than PostgresWorldStore in a release build.

  3. Create world is transactional for world state. worlds row and world_turns(turn_number=0) row are written together. This ticket emits no seed audit events. worlds.scenario_hash is a real foreign key into scenarios(hash). No "create then write back" two-step remains.

  4. Run-turn success is transactional. In one Postgres transaction: attempt status flipped to committed, new world_turns row inserted, all audit events for the turn inserted, all event-entity rows inserted, worlds.current_turn advanced, worlds.active_attempt_id cleared. All or none.

  5. Run-turn failure is transactional. In one Postgres transaction: attempt status flipped to failed, failure-related audit events inserted, worlds.active_attempt_id cleared, worlds.current_turn unchanged.

  6. Restart behavior is explicit. On binary startup, reconcile_running_attempts runs before the HTTP/MCP listener accepts traffic. Any running attempts become interrupted with a logged failure reason. Any worlds with active_attempt_id pointing at one of those have it cleared.

  7. Audit consumers are cursor-based. read_audit_events accepts AuditCursor; pagination is monotonic over world_event_seq. No LISTEN/NOTIFY is required by this ticket; if added later, it is a wake-up hint only, never the data channel.

  8. Component provenance is recorded at execution time. world_audit_events rows for perception_emitted, intent_formed, intent_adjudicated, and adjudication_rejected carry the relevant component hashes per the matrix in "Component hash provenance". Verified by a smoke step that runs a turn and SELECTs to confirm the hashes are present and match the scenario's profile components.

  9. Scenario summaries use real world counts. StoredScenario.world_count and ScenarioSummary.world_count are populated from worlds joined on scenario_hash, counting only status='active' worlds.

  10. Deletion is durable. delete_world flips status='deleted' and sets deleted_at. Default list_worlds excludes deleted; include_recently_deleted=true returns them. Busy worlds are rejected. No in-memory tombstone map exists in the production binary. Restart preserves deletion state.

  11. No DB transaction is held during LLM calls. Verified by code review: start_attempt, commit_turn, and fail_attempt are short transactions; the cognition step in Runtime::run_turn runs between them with no transaction handle in scope.

  12. No queued attempts exist. attempt_status enum has no queued value. run_turn returns an attempt in running state after start_attempt succeeds.

  13. Persisted turn state excludes cognition. world_turns.state serializes PersistedWorldState, not World. Tests prove cognition profiles and chronon_seconds are absent from the stored state and reattached from scenario content during reconstitution.

Cleanup grep guards

All MUST return zero matches in production src/ code after Phase F, excluding test code that asserts the absence of these symbols if any. The handler runs each as part of phase verification.

rg -n 'pub fn load_all' src/
rg -n 'WorldMeta::read\b' src/
rg -n 'WorldMeta::write_back' src/
rg -n 'pub scenario_snapshot' src/
rg -n 'struct DeletedWorldRecord' src/
rg -n 'mod persistence' src/
rg -n 'persistence::' src/
rg -n 'mod turn_job' src/
rg -n 'turn_job::' src/
rg -n 'attempts\.json' src/
rg -n 'audit/events\.jsonl' src/
rg -n 'turns/turn_' src/
rg -n '/var/lib/chukwa/worlds' src/
rg -n 'meta\.json' src/
rg -n '\.scenario_snapshot' src/
rg -n "'queued'" src/
rg -n 'AttemptStatus::Queued' src/
rg -n 'enqueue_attempt' src/
rg -n 'claim_attempt' src/

claim_attempt is intentionally absent because this ticket has a single start_attempt operation. If a future worker-queue ticket reintroduces claim semantics, it will add that deliberately.

Smoke plan

End-to-end live smoke against the deployed pod. Capture verbatim request/response for each step.

  1. Verify empty database. list_worlds returns count=0. Postgres:

    SELECT count(*) FROM worlds;
SELECT count(*) FROM world_turns;
SELECT count(*) FROM attempts;
SELECT count(*) FROM world_audit_events;
SELECT count(*) FROM world_audit_event_entities;

    All zero.

  2. Verify scenarios persisted. list_scenarios returns the existing scenarios from prior smokes, e.g. cat_in_library, vending-leak-fix, locked_vending_room. Their world_count is now actually computed and is 0 because no worlds exist yet.

  3. Create a world from a known scenario. create_world {slug: "smoke-world", scenario_ref: {name: "vending-leak-fix"}}. Response carries scenario_hash matching vending-leak-fix's manifest hash. Verify in Postgres:

    SELECT * FROM worlds WHERE slug='smoke-world';

    Row exists with status='active', current_turn=0, active_attempt_id=NULL. Then:

    SELECT * FROM world_turns
WHERE world_slug='smoke-world' AND turn_number=0;

    Row exists with attempt_id=NULL.

  4. Verify created_from_ref normalization. Postgres:

    SELECT created_from_ref FROM worlds WHERE slug='smoke-world';

    Shape is {kind:'name', input:'vending-leak-fix', resolved_hash:'...'}. It does not contain a scenario snapshot.

  5. Verify scenario world_count updated. list_scenarios filtered to vending-leak-fix: world_count=1.

  6. Run a turn. run_turn {world_slug: "smoke-world"} returns an attempt_id with status running. Poll get_turn_status. Observe running, then committed. Duration logged.

  7. Verify transactional commit. Postgres:

    SELECT status FROM attempts WHERE attempt_id = '<attempt_id>';
SELECT * FROM world_turns WHERE world_slug='smoke-world' AND turn_number=1;
SELECT current_turn, active_attempt_id FROM worlds WHERE slug='smoke-world';
SELECT count(*) FROM world_audit_events WHERE world_slug='smoke-world' AND turn_number=1;
SELECT * FROM world_audit_event_entities WHERE world_slug='smoke-world';

    Expected:

    • attempt status = committed
    • turn 1 row exists
    • worlds pointer = (1, NULL)
    • audit event count ≥ 4: perception, intent, adjudication, turn_complete
    • event-entity rows exist for the agent

  8. Verify component provenance. SELECT a perception_emitted event for turn 1: perceive_system_hash is non-NULL and equals vending-leak-fix's subject profile's perceive_system hash. Same for intent_formed.intend_system_hash, intent_adjudicated.adjudicate_system_hash, and intent_adjudicated.adjudication_schema_hash.

  9. Cursor pagination. read_audit_events or MCP equivalent get_events with cursor=0, limit=2 returns the first two events plus a next_cursor. Calling again with that cursor returns the next events.

  10. Entity history. entity_history(slug, "subject") returns audit events touching the subject entity, in order. Side-table query path verified.

  11. World list and detail. list_worlds returns smoke-world. get_world returns details with current_turn=1, attempt_count=1, recent last_activity.

  12. Busy delete rejection, if practical. Start a deliberately slow turn or use a test fixture to create a running attempt. delete_world returns Busy and does not clear active_attempt_id. If hard to drive live, this remains covered by postgres tests.

  13. Delete world. delete_world(slug, reason="smoke cleanup"). Postgres:

SELECT status, deleted_at, deleted_reason
FROM worlds
WHERE slug='smoke-world';

Expected: ('deleted', <now>, 'smoke cleanup'). World tables for that slug are NOT cleaned up; rows persist with status flipped. list_worlds default does not include it. list_worlds(include_recently_deleted=true) does.

  1. Verify scenario world_count after delete. list_scenarios: vending-leak-fix.world_count = 0 again because deleted worlds are excluded.

  2. Restart recovery test. Pre-flight: have a running attempt by either killing mid-cognition during a deliberately slow turn, or using a postgres-test fixture. After restart/reconcile: list_attempts for the affected world shows the attempt with status='interrupted' and failure_reason='process restart before commit'. worlds.active_attempt_id is NULL. The world can run a new turn successfully.

If step 15 is hard to drive in production, it can be satisfied by a postgres-tests-feature integration test that simulates the crash by leaving an attempt in running state and calling reconcile_running_attempts directly.

The smoke is considered green if steps 1–14 pass live and step 15 passes either live or in a postgres test.

Implementation guidance

Concurrency and locks

The single-writer-per-world rule is preserved. The existing per-world tokio Mutex stays. The DB lease (active_attempt_id) is belt-and-suspenders today; it becomes load-bearing if we ever go multi-process.

If we want multi-process safety later, out of scope for this ticket but worth designing toward, the start operation can become a compare-and-swap:

UPDATE worlds
SET active_attempt_id = $attempt_id
WHERE slug = $slug
  AND status = 'active'
  AND active_attempt_id IS NULL
RETURNING current_turn;

No row returned = lost the race. Multi-process workers picking from a queue would use FOR UPDATE SKIP LOCKED on a future queue table or on future queued attempts. Do not build that now; just do not preclude it.

JSON storage discipline

world_turns.state and world_audit_events.event are JSONB. Use them for opaque payload storage. Do not over-promise their queryability. Key-value queries via JSONB GIN indexes are possible but not the first-line query pattern.

The typed columns on world_audit_events are the supported query surface. If a future query needs a field not yet typed, add a column in a migration.

Migration ordering

Migration 0002 references scenarios(hash) and component tables from 0001. This is fine; sqlx applies migrations in order.

Do not force raw SQL idempotency by adding defensive IF NOT EXISTS to every DDL statement. The required idempotency is migration-runner idempotency: once 0002 has applied, a second runner invocation sees no pending migration and succeeds.

Error handling

WorldStoreError::CommitRaceLost should never fire under correct single-writer behavior. If it does in production, log loudly. If it ever shows up in tests, that is a real bug — likely a missed lock, stale attempt, or missing FOR UPDATE.

WorldStoreError::LeaseInvalid should likewise be rare. It means the attempt trying to commit/fail is not the world's active attempt. Treat it as a correctness failure, not a normal user error.

WorldStoreError::Database(String) wraps sqlx errors. Do not rely on its inner string for control flow. Use typed variants for things callers need to handle.

End of ticket.

Proposed resolution

Postgres-native world store is live in production. Worlds, attempts, turns, audit events, and execution provenance now live in Postgres; the file-backed substrate is gone.

Phase summary

PhaseCommitWhat landedDeployable
A2e74d0fschema 0002 + module skeletony
B4243e68PostgresWorldStore + MemoryWorldStore impl + 30 postgres-testsy
Cd96eee0kernel rewrite + bin startup wiring + reconcile_running_attemptsy
D82c57e6MCP handlers route through WorldStore + cursor pagination on get_eventsy
E0908c67web routes route through WorldStorey
Fb9ea61ddelete file-backed paths (persistence.rs, worlds::load_all, scenario_snapshot, tombstone map, etc.)y
G(operational, no commit)live worlds purged from prod, count=0n/a
Hc50454f8 (merge)merged feat/world-store-db → main, image rolled to chukwa-b9c5f699b-9k7jn, migration 0002 applied success=t, reconcile_running_attempts=0, live smoke 12/12 passedy

Test counts at completion

Delta vs start-of-ticket: lib went 420 → 407 because some file-backed-only tests were deleted in Phase F; new component-hash and store-trait tests were added across B/C. The net contraction is fine — the deleted tests covered the deleted code.

Live smoke evidence (Phase H)

Twelve-step smoke against the rolled pod, image chukwa-b9c5f699b-9k7jn:

  1. list_worlds — empty registry baseline
  2. list_scenarios — scenario catalogue intact
  3. create_world — fresh world minted from a scenario
  4. get_world — canonical state retrieved
  5. run_turn — real LLM-driven turn (perceive → intend → adjudicate → commit, ~20s)
  6. get_turn_status — attempt transitions to committed
  7. list_turns — committed chain visible
  8. get_turn w/ include_events — event payload returned with the snapshot
  9. get_events — cursor + pagination behaved as specified
  10. entity_history — subject side-table auto-emit returned all events touching the entity
  11. delete_world — world status flipped to deleted
  12. list_worlds — empty again

All 12/12 passed; full hash-chain integrity verified end-to-end.

Architectural delta

Surfaced for follow-up

(Suggestions only — not filed.)

Closing

All five Phase 0 axioms continue to hold. The substrate trajectory from 7d14ef0b (scenario store) to this ticket (world store) is complete; chukwa is now Postgres-native end-to-end with the in-process state surface gone. Awaiting caller acceptance.

History (18 events)

Sign in as a human to drive this ticket from the page, or use the MCP tools.