Guides
View as Markdown Open in ChatGPT Open in Claude

Pipeline State Store

The pipeline state store is a key/value store scoped to each pipeline run. Stages can read and write state entries to share data — analysis results, computed artifacts, configuration decisions — without committing files to git or using external storage.

Why State Store?

Before state store, multi-stage pipelines had awkward options for sharing data:

Pattern Problem
Commit to git branch Pollutes history, slow clone times, merge conflicts
External S3/Redis Extra infrastructure, credentials, cleanup
Environment variables Size limits, can't pass structured data
File system Lost when sandbox terminates

The state store provides:

  • Structured data — JSON, strings, file references, patches
  • Automatic cleanup — TTL-based expiration, no orphaned artifacts
  • Access control — Scoped to the run, no cross-pipeline leakage
  • Observable — State visible in dashboard, API, CLI
  • Size limits — Prevents unbounded growth (10 MB per run)

Core Concepts

State Entry

A key/value pair with metadata:

{
  "key": "analysis_results",
  "value": "{\"files\": [\"src/main.ts\"], \"score\": 85}",
  "type": "json",
  "mime_type": "application/json",
  "size_bytes": 42,
  "created_by": "analyze",
  "created_at": "2026-08-02T10:05:30Z",
  "expires_at": "2026-08-09T10:05:30Z"
}

Value Types

Type Description Use Case
string Plain text Status flags, IDs, short messages
json JSON object/array Analysis results, configurations, lists
file File path reference Build artifacts, generated files
patch Git diff format Code changes for review
artifact Binary blob reference Compiled outputs, images

TTL (Time-to-Live)

State entries expire automatically:

  • Default TTL: 7 days
  • Maximum TTL: 30 days
  • Custom TTL: Set per-entry when writing

Expired entries are cleaned up automatically — no manual garbage collection needed.

Usage Patterns

Writing State from Stages

Agent stages can write state using the caged state command available inside sandboxes:

stages:
  - name: analyze
    type: agent
    agent: claude-code
    prompt: |
      Analyze the codebase for refactoring opportunities.
      
      When done, save your findings:
      caged state set analysis '{"files": [...], "priority": "high"}'

Command stages can also write state:

stages:
  - name: compute-hash
    type: command
    command: |
      HASH=$(sha256sum ./dist/app.js | cut -d' ' -f1)
      caged state set build_hash "$HASH"

Reading State in Subsequent Stages

Later stages read state to continue the work:

stages:
  - name: refactor
    type: agent
    agent: claude-code
    prompt: |
      Read the analysis results and refactor the identified files.
      
      First, get the analysis:
      caged state get analysis
      
      Then refactor each file listed in the results.
    depends_on: [analyze]

Multi-Agent Handoff

The most powerful pattern — multiple specialized agents collaborate through state:

pipelines:
  - name: ai-code-review
    stages:
      # Agent 1: Implements the feature
      - name: implement
        type: agent
        agent: claude-code
        prompt: |
          Implement: $FEATURE_SPEC
          
          When complete, summarize what you did:
          caged state set implementation '{
            "files_changed": ["src/feature.ts", "src/feature.test.ts"],
            "approach": "Used factory pattern for extensibility",
            "test_coverage": 87
          }' --type json
        budget: 15.00

      # Agent 2: Security review (reads implementation context)
      - name: security-review
        type: agent
        agent: claude-code
        prompt: |
          Review the implementation for security issues.
          
          Get context:
          caged state get implementation
          
          Save your findings:
          caged state set security_findings '{
            "issues": [...],
            "severity": "low",
            "approved": true
          }' --type json
        depends_on: [implement]
        budget: 5.00

      # Agent 3: Performance review (parallel with security)
      - name: perf-review
        type: agent
        agent: aider
        prompt: |
          Review for performance issues.
          
          caged state get implementation
          caged state set perf_findings '{"issues": [...]}' --type json
        depends_on: [implement]
        budget: 5.00

      # Final stage: Compile all findings
      - name: compile-report
        type: command
        command: |
          impl=$(caged state get implementation)
          security=$(caged state get security_findings)
          perf=$(caged state get perf_findings)
          
          echo "{
            \"implementation\": $impl,
            \"security\": $security,
            \"performance\": $perf
          }" | jq '.' > report.json
          
          caged state set final_report -f report.json --type json
        depends_on: [security-review, perf-review]

DAG visualization:

           implement
          /         \
   security-review  perf-review
          \         /
        compile-report

Each agent sees only what it needs, writes its findings, and the final stage aggregates everything.

Incremental Processing

For long-running migrations, track progress in state:

stages:
  - name: scan
    type: command
    command: |
      # Find all files to migrate
      find src -name "*.js" > /tmp/files.txt
      caged state set remaining "$(cat /tmp/files.txt | jq -R . | jq -s .)" --type json
      caged state set completed "[]" --type json

  - name: migrate-batch
    type: agent
    agent: claude-code
    prompt: |
      Get the migration state:
      remaining=$(caged state get remaining)
      completed=$(caged state get completed)
      
      Process 10 files from 'remaining'.
      Move processed files to 'completed'.
      Update state when done.
      
      If all files are migrated, write:
      caged state set migration_complete "true"
    depends_on: [scan]
    timeout: 30m

  - name: continue-check
    type: gate
    depends_on: [migrate-batch]
    gate:
      # Continue if migration not complete
      if: "state['migration_complete'] != 'true'"

Human Approval with Context

Pass context to human reviewers through state:

stages:
  - name: implement
    type: agent
    agent: claude-code
    prompt: |
      Implement the feature and save a summary for human review:
      caged state set review_summary '{
        "what_changed": "Added user preferences API",
        "files": 5,
        "test_coverage": 92,
        "breaking_changes": false
      }' --type json

  - name: approve
    type: await_approval
    depends_on: [implement]
    approval:
      message: |
        Agent completed implementation. 
        
        Review summary available at:
        caged pipeline state get $PIPELINE_ID $RUN_ID review_summary
      channels: [slack, dashboard]

CLI Usage

Outside the Sandbox

Manage state from your local machine:

# List all state entries
caged pipeline state list pipe-abc123 run-xyz789

# Get a specific entry
caged pipeline state get pipe-abc123 run-xyz789 analysis_results

# Set a value
caged pipeline state set pipe-abc123 run-xyz789 config '{"debug": true}' --type json

# Set from file
caged pipeline state set pipe-abc123 run-xyz789 report -f ./report.json --type json --ttl 86400

# Delete an entry
caged pipeline state delete pipe-abc123 run-xyz789 temp_data

Inside the Sandbox

Within a pipeline stage, use the simpler form (run context is implicit):

# These work inside pipeline sandboxes
caged state set my_key "my_value"
caged state get my_key
caged state list

API Usage

Full CRUD via REST API:

# List state
curl https://api.caged.dev/v1/pipelines/{id}/runs/{runId}/state \
  -H "Authorization: Bearer caged_sk_..."

# Get single entry
curl https://api.caged.dev/v1/pipelines/{id}/runs/{runId}/state/my_key \
  -H "Authorization: Bearer caged_sk_..."

# Set entry
curl -X PUT https://api.caged.dev/v1/pipelines/{id}/runs/{runId}/state/my_key \
  -H "Authorization: Bearer caged_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "value": {"result": "success"},
    "type": "json",
    "ttl_seconds": 86400
  }'

# Delete entry
curl -X DELETE https://api.caged.dev/v1/pipelines/{id}/runs/{runId}/state/my_key \
  -H "Authorization: Bearer caged_sk_..."

MCP Tools

When using the Caged MCP server, four state tools are available:

Tool Description
pipeline_state_list List all state entries for a run
pipeline_state_get Get a single state entry
pipeline_state_set Create or update a state entry
pipeline_state_delete Delete a state entry

See MCP Server Reference for full tool documentation.

Limits

Limit Value Rationale
Max key length 256 characters Prevent key abuse
Max value size 1 MB Keep state fast, use objstore for large files
Max entries per run 100 Prevent key explosion
Max total size per run 10 MB Bound memory/storage
Default TTL 7 days Auto-cleanup
Max TTL 30 days Prevent permanent storage

For artifacts larger than 1 MB, store them in object storage and write the URL/path to state.

Best Practices

  1. Use descriptive keys — security_review_findings not data1
  2. Set appropriate TTL — Short TTL for temp data, longer for audit trails
  3. Use JSON for structured data — Easier to parse in subsequent stages
  4. Include metadata — Timestamp, creator, version in your JSON values
  5. Don't store secrets — State is visible in dashboard/API
  6. Clean up when done — Delete temp entries to stay under limits
Was this page helpful?