Sign in to edit tickets from this page.

← all tickets · home

Add multi-turn `run_turn` support with durable turn-run lifecycle

resolved 0cac6740-2728-4e29-acec-5047e25f23f4

created_at
2026-05-01
updated_at
2026-05-01
priority
P1
ticket_type
feature
resolved_at
2026-05-01
resolution
accepted

Body

The key design decision: keep one attempt per attempted turn. That preserves the current lifecycle model, LLM trace visibility, failure records, list_attempts, get_turn_status, and canonical turn history. Add a higher-level “turn run” / “series” wrapper that sequences those attempts.

The relevant pieces are:


Ticket: Add multi-turn run_turn support with durable turn-run lifecycle

Goal

Extend the MCP run_turn tool so callers can request more than one committed turn in a single tool call by passing an optional integer parameter named turn_count.

The implementation must preserve the existing per-turn attempt lifecycle. A multi-turn run must create and execute one normal attempt per attempted turn. The system must expose durable visibility into the multi-turn run, must allow cancellation between attempts, and must keep every individual attempt observable through the existing attempt lifecycle tools.

Non-goals

Do not change cognition workflow semantics.

Do not change WorldPatch.

Do not change how a single turn is committed or failed.

Do not create a single attempt that commits multiple turns.

Do not remove or weaken the existing attempts lifecycle.

Do not make multi-turn execution parallel. Multi-turn runs are strictly sequential per world.


Required public tool behavior

run_turn

Update the existing run_turn MCP tool to accept these arguments:

{
  "world_slug": "string",
  "turn_count": "optional integer",
  "max_attempts": "optional integer"
}

turn_count rules:

max_attempts rules:

The tool must reject unknown keys. The accepted keys for run_turn are exactly:

world_slug
turn_count
max_attempts

Single-attempt compatibility mode

When turn_count is omitted and max_attempts is omitted, preserve the current behavior:

The response must add these fields:

{
  "run_mode": "single_attempt",
  "turn_count": 1,
  "turn_count_source": "default",
  "turn_count_hint": "No turn_count was supplied; run_turn defaulted to turn_count=1 and started one single-turn attempt.",
  "max_attempts": 1,
  "max_attempts_source": "default",
  "max_attempts_hint": "No max_attempts was supplied; max_attempts defaulted to turn_count (1)."
}

The existing fields must remain:

{
  "world_slug": "...",
  "attempt_id": "...",
  "status": "running",
  "turn_before": 123,
  "attempted_turn": 124,
  "poll_with": {
    "tool": "get_turn_status",
    "args": {
      "world_slug": "...",
      "attempt_id": "..."
    }
  }
}

Explicit single-turn mode

When turn_count is explicitly supplied as 1 and max_attempts is omitted or explicitly 1, use the same single-attempt path.

Set:

{
  "run_mode": "single_attempt",
  "turn_count": 1,
  "turn_count_source": "explicit",
  "turn_count_hint": "turn_count was supplied as 1; run_turn started one single-turn attempt."
}

Multi-turn mode

When turn_count > 1, or when max_attempts > 1, run_turn must create a durable turn-run row and spawn a background coordinator task.

The response must include:

{
  "run_mode": "turn_run",
  "world_slug": "...",
  "turn_run_id": "...",
  "status": "running",
  "turn_count": 40,
  "turn_count_source": "explicit",
  "turn_count_hint": "turn_count was supplied as 40; run_turn started a turn run targeting 40 committed turn(s).",
  "max_attempts": 40,
  "max_attempts_source": "default",
  "max_attempts_hint": "No max_attempts was supplied; max_attempts defaulted to turn_count (40).",
  "start_turn": 123,
  "target_turn": 163,
  "poll_with": {
    "tool": "get_turn_run_status",
    "args": {
      "world_slug": "...",
      "turn_run_id": "..."
    }
  },
  "list_attempts_with": {
    "tool": "list_attempts",
    "args": {
      "world_slug": "...",
      "turn_run_id": "..."
    }
  }
}

For explicit max_attempts, the hint must be:

max_attempts was supplied as {max_attempts}; the turn run will stop after at most {max_attempts} attempt(s).

Attempt semantics

A turn run must not pre-create attempts.

A turn run must create attempts lazily, one at a time.

Each attempt must still represent exactly one attempted turn.

A successful attempt increments world current_turn by exactly one.

A failed attempt does not increment world current_turn.

The turn run must continue creating attempts until one of these terminal conditions is reached:

  1. committed_turn_count == requested_turn_count

    • Turn run status becomes completed.
  2. attempt_count == max_attempts and committed_turn_count < requested_turn_count

    • Turn run status becomes failed.

    • Failure reason must be exactly:

      max_attempts exhausted before requested turn_count committed
      
  3. Cancellation was requested and the current active attempt has finished.

    • Turn run status becomes cancelled.
  4. The process restarts before the turn run completes.

    • Startup reconciliation marks the turn run interrupted.

A failed individual attempt must remain visible as a normal failed attempt through get_turn_status and list_attempts.

A failed individual attempt must not automatically fail the whole turn run unless max_attempts has been exhausted.


New public tools

Add two consumer MCP tools.

get_turn_run_status

Input:

{
  "world_slug": "string",
  "turn_run_id": "uuid",
  "include_attempts": "optional boolean",
  "attempt_limit": "optional integer"
}

Rules:

Return:

{
  "message": "...",
  "world_slug": "...",
  "turn_run_id": "...",
  "status": "running",
  "requested_turn_count": 1000,
  "max_attempts": 3000,
  "start_turn": 20,
  "target_turn": 1020,
  "current_turn": 25,
  "committed_turn_count": 5,
  "remaining_committed_turns": 995,
  "attempt_count": 3000,
  "failed_attempt_count": 2995,
  "interrupted_attempt_count": 0,
  "active_attempt_id": null,
  "last_attempt_id": "...",
  "last_attempt_status": "failed",
  "progress": "...",
  "cancel_requested_at": null,
  "cancel_reason": null,
  "enqueued_at": "...",
  "started_at": "...",
  "ended_at": null,
  "poll_active_attempt_with": null,
  "list_attempts_with": {
    "tool": "list_attempts",
    "args": {
      "world_slug": "...",
      "turn_run_id": "..."
    }
  }
}

When include_attempts is true, include:

{
  "recent_attempts": [
    {
      "attempt_id": "...",
      "turn_run_id": "...",
      "turn_run_seq": 1,
      "status": "committed",
      "turn_before": 20,
      "attempted_turn": 21,
      "produced_turn": 21
    }
  ]
}

The recent attempts must be ordered newest first.

cancel_turn_run

Input:

{
  "world_slug": "string",
  "turn_run_id": "uuid",
  "reason": "optional string"
}

Behavior:

Return the same status shape as get_turn_run_status.


Existing public tool changes

get_turn_status

Augment the existing attempt response with:

{
  "turn_run_id": null,
  "turn_run_seq": null
}

For attempts that belong to a turn run, these fields must be populated.

list_attempts

Accept optional turn_run_id.

Current accepted keys become:

world_slug
turn_run_id

When turn_run_id is provided, list only attempts belonging to that turn run.

Every attempt summary must include:

{
  "turn_run_id": null,
  "turn_run_seq": null
}

Database migration

Add migration:

migrations/0008_turn_runs.sql

Create enum:

CREATE TYPE turn_run_status AS ENUM (
    'running',
    'cancel_requested',
    'completed',
    'failed',
    'cancelled',
    'interrupted'
);

Alter worlds:

ALTER TABLE worlds
    ADD COLUMN active_turn_run_id UUID;

Create table:

CREATE TABLE turn_runs (
    turn_run_id UUID PRIMARY KEY,
    world_slug label_text NOT NULL REFERENCES worlds(slug),

    status turn_run_status NOT NULL,

    enqueued_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    ended_at TIMESTAMPTZ,

    worker_id TEXT NOT NULL,

    start_turn BIGINT NOT NULL CHECK (start_turn >= 0),
    target_turn BIGINT NOT NULL CHECK (target_turn >= 1),

    requested_turn_count INT NOT NULL CHECK (requested_turn_count >= 1),
    max_attempts INT NOT NULL CHECK (max_attempts >= 1),

    turn_count_source TEXT NOT NULL CHECK (turn_count_source IN ('default', 'explicit')),
    max_attempts_source TEXT NOT NULL CHECK (max_attempts_source IN ('default', 'explicit')),

    committed_turn_count INT NOT NULL DEFAULT 0 CHECK (committed_turn_count >= 0),
    attempt_count INT NOT NULL DEFAULT 0 CHECK (attempt_count >= 0),
    failed_attempt_count INT NOT NULL DEFAULT 0 CHECK (failed_attempt_count >= 0),
    interrupted_attempt_count INT NOT NULL DEFAULT 0 CHECK (interrupted_attempt_count >= 0),

    active_attempt_id UUID,
    last_attempt_id UUID,

    progress TEXT,
    cancel_requested_at TIMESTAMPTZ,
    cancel_reason TEXT,
    failure_reason TEXT,

    CONSTRAINT turn_runs_target_check
        CHECK (target_turn = start_turn + requested_turn_count),

    CONSTRAINT turn_runs_max_attempts_check
        CHECK (max_attempts >= requested_turn_count),

    CONSTRAINT turn_runs_terminal_time_check
        CHECK (
            (status IN ('running', 'cancel_requested') AND ended_at IS NULL)
            OR
            (status IN ('completed', 'failed', 'cancelled', 'interrupted') AND ended_at IS NOT NULL)
        ),

    CONSTRAINT turn_runs_cancel_check
        CHECK (
            (status IN ('cancel_requested', 'cancelled') AND cancel_requested_at IS NOT NULL)
            OR
            (status NOT IN ('cancel_requested', 'cancelled'))
        ),

    CONSTRAINT turn_runs_failure_check
        CHECK (
            (status IN ('failed', 'interrupted') AND failure_reason IS NOT NULL)
            OR
            (status NOT IN ('failed', 'interrupted'))
        ),

    CONSTRAINT turn_runs_world_run_unique UNIQUE (world_slug, turn_run_id)
);

Indexes:

CREATE INDEX turn_runs_world_enqueued_idx
    ON turn_runs(world_slug, enqueued_at DESC);

CREATE INDEX turn_runs_world_status_idx
    ON turn_runs(world_slug, status);

CREATE UNIQUE INDEX turn_runs_one_active_per_world_idx
    ON turn_runs(world_slug)
    WHERE status IN ('running', 'cancel_requested');

Alter attempts:

ALTER TABLE attempts
    ADD COLUMN turn_run_id UUID,
    ADD COLUMN turn_run_seq INT;

ALTER TABLE attempts
    ADD CONSTRAINT attempts_turn_run_pair_check
    CHECK (
        (turn_run_id IS NULL AND turn_run_seq IS NULL)
        OR
        (turn_run_id IS NOT NULL AND turn_run_seq IS NOT NULL AND turn_run_seq >= 1)
    );

ALTER TABLE attempts
    ADD CONSTRAINT attempts_turn_run_fk
    FOREIGN KEY (world_slug, turn_run_id)
    REFERENCES turn_runs(world_slug, turn_run_id);

CREATE UNIQUE INDEX attempts_turn_run_seq_idx
    ON attempts(turn_run_id, turn_run_seq)
    WHERE turn_run_id IS NOT NULL;

CREATE INDEX attempts_turn_run_idx
    ON attempts(turn_run_id, turn_run_seq);

Add deferred FKs after both tables exist:

ALTER TABLE turn_runs
    ADD CONSTRAINT turn_runs_active_attempt_fk
    FOREIGN KEY (world_slug, active_attempt_id)
    REFERENCES attempts(world_slug, attempt_id)
    DEFERRABLE INITIALLY DEFERRED;

ALTER TABLE turn_runs
    ADD CONSTRAINT turn_runs_last_attempt_fk
    FOREIGN KEY (world_slug, last_attempt_id)
    REFERENCES attempts(world_slug, attempt_id)
    DEFERRABLE INITIALLY DEFERRED;

ALTER TABLE worlds
    ADD CONSTRAINT worlds_active_turn_run_fk
    FOREIGN KEY (slug, active_turn_run_id)
    REFERENCES turn_runs(world_slug, turn_run_id)
    DEFERRABLE INITIALLY DEFERRED;

Store-layer changes

Add these public DTOs to src/world_store/mod.rs:

pub struct TurnRunId(pub Uuid);

pub enum TurnRunStatus {
    Running,
    CancelRequested,
    Completed,
    Failed,
    Cancelled,
    Interrupted,
}

pub struct StartTurnRunInput {
    pub world_slug: Slug,
    pub worker_id: String,
    pub requested_turn_count: u32,
    pub max_attempts: u32,
    pub turn_count_source: String,
    pub max_attempts_source: String,
}

pub struct TurnRunRecord {
    pub turn_run_id: TurnRunId,
    pub world_slug: Slug,
    pub status: TurnRunStatus,
    pub requested_turn_count: u32,
    pub max_attempts: u32,
    pub start_turn: u64,
    pub target_turn: u64,
    pub committed_turn_count: u32,
    pub attempt_count: u32,
    pub failed_attempt_count: u32,
    pub interrupted_attempt_count: u32,
    pub active_attempt_id: Option<AttemptId>,
    pub last_attempt_id: Option<AttemptId>,
    pub progress: Option<String>,
    pub cancel_requested_at: Option<DateTime<Utc>>,
    pub cancel_reason: Option<String>,
    pub failure_reason: Option<String>,
    pub enqueued_at: DateTime<Utc>,
    pub started_at: DateTime<Utc>,
    pub ended_at: Option<DateTime<Utc>>,
}

Add these methods to WorldStore:

async fn start_turn_run(
    &self,
    input: StartTurnRunInput,
) -> Result<TurnRunRecord, WorldStoreError>;

async fn start_attempt_for_turn_run(
    &self,
    turn_run_id: TurnRunId,
    worker_id: &str,
) -> Result<ClaimedAttempt, WorldStoreError>;

async fn finish_turn_run_attempt(
    &self,
    turn_run_id: TurnRunId,
    attempt_id: AttemptId,
) -> Result<TurnRunRecord, WorldStoreError>;

async fn get_turn_run_status(
    &self,
    turn_run_id: TurnRunId,
) -> Result<TurnRunRecord, WorldStoreError>;

async fn request_turn_run_cancel(
    &self,
    turn_run_id: TurnRunId,
    reason: String,
) -> Result<TurnRunRecord, WorldStoreError>;

async fn finalize_cancelled_turn_run_if_idle(
    &self,
    turn_run_id: TurnRunId,
) -> Result<TurnRunRecord, WorldStoreError>;

The exact method names above must be used.

start_turn_run

This method must:

  1. Lock the world row.
  2. Reject deleted or unknown worlds using existing errors.
  3. Reject if active_attempt_id is not null.
  4. Reject if active_turn_run_id is not null.
  5. Insert a turn_runs row with status running.
  6. Set worlds.active_turn_run_id.
  7. Return the TurnRunRecord.

Existing start_attempt

Update existing start_attempt so it rejects when worlds.active_turn_run_id is not null.

The error must be WorldStoreError::Busy.

The returned busy attempt id string should be the active_turn_run_id string when no active attempt exists.

start_attempt_for_turn_run

This method must:

  1. Lock the turn run row.
  2. Lock the world row.
  3. Verify the turn run status is running.
  4. Verify worlds.active_turn_run_id == turn_run_id.
  5. Verify worlds.active_attempt_id IS NULL.
  6. Verify attempt_count < max_attempts.
  7. Load the current turn state and scenario exactly like start_attempt.
  8. Insert a normal attempts row.
  9. Populate attempts.turn_run_id.
  10. Populate attempts.turn_run_seq = turn_runs.attempt_count + 1.
  11. Set worlds.active_attempt_id.
  12. Set turn_runs.active_attempt_id.
  13. Set turn_runs.last_attempt_id.
  14. Increment turn_runs.attempt_count.
  15. Return ClaimedAttempt.

finish_turn_run_attempt

This method must:

  1. Lock the turn run row.

  2. Lock the attempt row.

  3. Verify the attempt belongs to the turn run.

  4. Verify the attempt is terminal: committed, failed, or interrupted.

  5. Clear turn_runs.active_attempt_id if it equals the attempt id.

  6. Increment counters:

    • committed_turn_count += 1 if attempt status is committed
    • failed_attempt_count += 1 if attempt status is failed
    • interrupted_attempt_count += 1 if attempt status is interrupted
  7. If committed_turn_count == requested_turn_count, set status completed, set ended_at, clear worlds.active_turn_run_id.

  8. Else if status is cancel_requested, set status cancelled, set ended_at, clear worlds.active_turn_run_id.

  9. Else if attempt_count >= max_attempts, set status failed, set ended_at, set failure reason exactly:

    max_attempts exhausted before requested turn_count committed
    

    and clear worlds.active_turn_run_id.

  10. Else leave status running.

request_turn_run_cancel

This method must:

  1. Lock the turn run row.

  2. If status is terminal, return the row without mutation.

  3. Set cancel_requested_at and cancel_reason.

  4. If active_attempt_id is null:

    • set status cancelled
    • set ended_at
    • clear worlds.active_turn_run_id
  5. If active_attempt_id is not null:

    • set status cancel_requested
    • do not mutate the active attempt

reconcile_running_attempts

Update startup reconciliation so it also handles turn runs.

After existing running attempts are marked interrupted, mark every turn_runs row with status running or cancel_requested as interrupted.

Set failure reason exactly:

process restart before turn run completed

Set ended_at.

Clear worlds.active_turn_run_id for affected worlds.

Also clear turn_runs.active_attempt_id.


Runtime changes

Add a background coordinator in src/kernel.rs or a new module src/turn_runs.rs.

The coordinator must:

  1. Load the turn run record.
  2. Stop if status is terminal.
  3. Stop if cancellation is requested.
  4. Stop if committed count reached target.
  5. Stop if max attempts reached.
  6. Claim one attempt through start_attempt_for_turn_run.
  7. Run that attempt through Runtime::run_claimed.
  8. Regardless of success or failure, call finish_turn_run_attempt.
  9. Loop.

The coordinator must not call Runtime::run_turn, because Runtime::run_turn uses start_attempt, and start_attempt must reject while a turn run holds the world.

The coordinator must call Runtime::run_claimed with the ClaimedAttempt returned by start_attempt_for_turn_run.

When Runtime::run_claimed returns an error caused by a normal failed attempt, the coordinator must still call finish_turn_run_attempt and continue if the turn run is still eligible to continue.

When the coordinator itself encounters a store error that prevents progress, mark the turn run failed, set failure_reason to the error string, set ended_at, and clear worlds.active_turn_run_id.


MCP implementation changes

Tool registry

Add these tools to CONSUMER_TOOLS and ALL_TOOLS:

get_turn_run_status
cancel_turn_run

Keep the tool partition test passing.

Tool schemas

Update consumer_tool_contract:

Update validate_tool_args / unknown-key rejection accordingly.

handle_run_turn

Implement this exact branch structure:

  1. Parse world_slug.

  2. Parse turn_count, default 1.

  3. Parse max_attempts, default turn_count.

  4. Validate:

    • turn_count >= 1
    • turn_count <= 100000
    • max_attempts >= 1
    • max_attempts <= 1000000
    • max_attempts >= turn_count
  5. If turn_count == 1 and max_attempts == 1, run the existing single-attempt path.

  6. Otherwise:

    • call world_store.start_turn_run
    • spawn the turn-run coordinator task
    • return the multi-turn response shape described above

handle_get_turn_run_status

Return the status shape described above.

When active_attempt_id is present, include:

{
  "poll_active_attempt_with": {
    "tool": "get_turn_status",
    "args": {
      "world_slug": "...",
      "attempt_id": "..."
    }
  }
}

When active_attempt_id is null, poll_active_attempt_with must be null.

handle_cancel_turn_run

Call world_store.request_turn_run_cancel.

Return the same status shape as get_turn_run_status.

store_attempt_to_json

Include:

{
  "turn_run_id": null,
  "turn_run_seq": null
}

Populate when present.


Tests

Add and update tests in src/mcp/tests.rs, src/world_store/memory.rs, and src/world_store/postgres.rs.

Required MCP tests

  1. run_turn_without_turn_count_preserves_single_attempt_contract

    • Call run_turn with only world_slug.
    • Assert response has run_mode: "single_attempt".
    • Assert response has attempt_id.
    • Assert response has no turn_run_id.
    • Assert turn_count_source: "default".
  2. run_turn_with_turn_count_starts_turn_run

    • Call run_turn with turn_count: 3.
    • Assert response has run_mode: "turn_run".
    • Assert response has turn_run_id.
    • Assert response has no attempt_id.
    • Poll get_turn_run_status until terminal.
    • Assert final status completed.
    • Assert world advanced by 3 committed turns.
  3. run_turn_with_explicit_one_uses_single_attempt_path

    • Call run_turn with turn_count: 1.
    • Assert response has run_mode: "single_attempt".
    • Assert turn_count_source: "explicit".
  4. run_turn_rejects_invalid_turn_count

    • turn_count: 0 rejected.
    • turn_count: 100001 rejected.
    • max_attempts < turn_count rejected.
    • unknown keys rejected.
  5. get_turn_run_status_reports_active_and_recent_attempts

    • Start a multi-turn run.
    • Poll status.
    • Assert counters, active/last attempt fields, and list_attempts_with.
  6. cancel_turn_run_stops_before_next_attempt

    • Start a long run.
    • Call cancel_turn_run.
    • Assert status becomes cancel_requested or cancelled.
    • Assert no additional attempts are started after cancellation is observed.
    • Assert final status cancelled.
  7. list_attempts_filters_by_turn_run_id

    • Start a turn run.
    • Assert attempts returned all have that turn_run_id.

Required store tests

  1. start_turn_run_claims_world_run_lease

    • World gets active_turn_run_id.
    • Single start_attempt is rejected while turn run is active.
  2. start_attempt_for_turn_run_creates_attempt_with_sequence

    • Attempts have turn_run_id.
    • Attempts have sequential turn_run_seq.
  3. finish_turn_run_attempt_completes_on_requested_commits

    • After requested committed attempts, status is completed.
    • worlds.active_turn_run_id cleared.
  4. finish_turn_run_attempt_fails_on_max_attempts

    • Failed attempts increment failed counter.
    • Status becomes failed when max attempts exhausted.
    • Failure reason matches required text.
  5. request_turn_run_cancel_is_idempotent

    • Repeated cancel calls do not corrupt counters.
  6. reconcile_running_attempts_interrupts_turn_runs

    • A running turn run is marked interrupted on reconcile.
    • worlds.active_turn_run_id is cleared.

Required manifest tests

Update consumer_tool_manifest_schemas_match_handler_contracts.

Update consumer_manifest_examples_execute so the examples cover:


Acceptance criteria

This ticket is complete when all of the following are true:

  1. run_turn({"world_slug": "x"}) still starts exactly one attempt and returns an attempt_id.
  2. run_turn({"world_slug": "x", "turn_count": 40}) starts one durable turn run and returns a turn_run_id.
  3. Multi-turn execution creates one normal attempt per attempted turn.
  4. Individual attempts remain visible through get_turn_status and list_attempts.
  5. get_turn_run_status reports committed count, attempt count, failed count, interrupted count, active attempt, last attempt, and terminal status.
  6. cancel_turn_run prevents the turn run from starting another attempt after the active attempt finishes.
  7. Startup reconciliation cannot leave a world stuck with an active turn run.
  8. Existing one-turn callers remain compatible.
  9. The tool response includes deterministic hint fields stating whether turn_count and max_attempts were supplied or defaulted.
  10. All Postgres, memory-store, MCP, and manifest tests pass.

Proposed resolution

Revised cleanup is implemented, committed, pushed, deployed, and live-smoked.

Compliance matrix by review item:

Files changed:

New regression tests added:

Verification run:

Deploy and artifact receipts:

Live smoke against https://chukwa.benac.dev/mcp:

History (7 events)

Sign in as a human to drive this ticket from the page, or use the MCP tools.