Lab L15 — Governed Multi-Harness Capstone System (Pure Go)

Phase 11 · CAPSTONE PROJECT — Lab L15
Status: Authored & Empirically Verified.
Student Lab Package: Authenticated Direct Download from VTAlgo Platform (hefesto-lab15-capstone.zip)
Branches: main (starter template) · solution (reference architecture).
Canonical Path: Student repo only (hefesto-lab15-capstone) — not E:\bridle, not the live product.

1. Laboratory Objective

Construct from scratch in pure Go standard library (zero external dependencies in go.mod) an enterprise-grade Governed Multi-Agent / Multi-Harness System, demonstrating how orthogonal subsystems, cryptographic ledgers, deterministic failure containment, two-phase budget circuit breakers, and heterogeneous harness adapters deliver mission-critical reliability and resilience under adversarial conditions.

The student implements:

• A strongly typed domain model (pkg/domain) defining Model Tiers (Frontier, Standard, Fast), Agent Roles (Planner, Executor, Verifier), Authority Tiers (Authoritative, Admitted, Retrieved, Conjectural), Evidence items, and Handoff Payloads.
• An authoritative state engine (pkg/state) protected by sync.RWMutex, monotonically increasing version clocks, snapshot isolation, and an append-only SHA-256 Merkle event ledger with mathematical tamper detection (VerifyIntegrity()).
• A scoped, multi-tenant epistemic memory store (pkg/memory) with cryptographic content digests and provenance tracking.
• A high-performance Deterministic Inconsistency Detector (pkg/detector) evaluating candidate records against active canonical state in $\mathcal{O}(1)$ time, intercepting stale, false, conflicting, cross-scope, and authority-inverting memories.
• A DCRA Containment Engine (pkg/containment) enforcing non-destructive isolation via a QuarantineBuffer with immediate operational authority revocation (OperationalAuthorityRevoked = true), canonical state fallback injection (0.0% context contamination), and immutable SHA-256 sealed audit receipts.
• A real-time micro-dollar Budget Governor (pkg/budget) executing a two-phase commit (Reserve/Settle) with calibrated rate cards and hard atomic circuit breakers throwing ErrBudgetExceeded.
• A Heterogeneous Harness Adapter Layer (pkg/harness) exposing the normalized AgentHarness interface across both in-process Go native execution (NativeHarness) and isolated CLI child processes (SubprocessCLIAdapter).
• A Governance Kernel (pkg/governance) implementing Human-in-the-Loop (HITL) cryptographic approval gates and strictly typed multi-role handoff validation that segregates probabilistic model inference from verified oracle evidence.
• A Stateless Recovery Engine (pkg/recovery) validating the *Amnesic Agent Test*, reconstructing 100% of operational state from Merkle checkpoints with zero prior conversational memory.
• An Equal-Compute Benchmark Suite (pkg/benchmark) comparing Single-Agent Systems (SAS) against Multi-Agent Systems (MAS) under an identical \$0.05 budget.
• A certification metrics engine (pkg/metrics) quantifying Fault Containment Effectiveness (FCE = 100.0%), MTTR (< 300 µs), and Context Contamination (0.0%).
• A unified CLI utility (cmd/capstone-cli) with five operational modes: nominal, inject-failure, stateless-recovery, benchmark, and certify.
• A comprehensive 17/17 automated test suite passing in $<250$ ms with zero race conditions under go test -v -race ./tests.

2. System Architecture

┌──────────────────────────────────────────────────────────────────────────┐
│          HEFESTO GOVERNED MULTI-HARNESS CAPSTONE ENGINE (PURE GO)        │
│                                                                          │
│  ┌─────────────────────────┐           ┌──────────────────────────────┐  │
│  │ 1. Canonical State      │           │ 2. Epistemic Memory Store    │  │
│  │    sync.RWMutex Engine  │           │    Multi-Tenant Scoping      │  │
│  │    Monotonic Clocks     │           │    Cryptographic Digests     │  │
│  │    SHA-256 Merkle Log   │           │    TierHistorical Storage    │  │
│  └───────────┬─────────────┘           └──────────────┬───────────────┘  │
│              │                                        │                  │
│              ▼                                        ▼                  │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │ 3. Deterministic Inconsistency Detector                            │  │
│  │    Evaluates candidate memories against active canonical state     │  │
│  │    Intercepts: Stale · False · Conflicting · Scope · Inversion     │  │
│  └───────────────────────────────────┬────────────────────────────────┘  │
│                                      │                                   │
│              ┌───────────────────────┴───────────────────────┐           │
│              │ Anomaly Detected                              │ Clean     │
│              ▼                                               ▼           │
│  ┌─────────────────────────────────────┐         ┌────────────────────┐  │
│  │ 4. DCRA Quarantine Engine           │         │ 5. Normal Path     │  │
│  │    • Non-destructive Isolation      │         │    • Admitted      │  │
│  │    • Authority Revoked = true       │         │    • Promoted to   │  │
│  │    • Canonical Fallback Injected    │         │      Prompt Buffer │  │
│  │    • SHA-256 Sealed Audit Receipt   │         │    • Zero Leakage  │  │
│  └──────────────────┬──────────────────┘         └─────────┬──────────┘  │
│                     │                                      │             │
│                     └───────────────────┬──────────────────┘             │
│                                         │                                │
│                                         ▼                                │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │ 6. Governance & Control Plane                                      │  │
│  │    • Two-Phase Micro-Dollar Budget Fencing (Reserve / Settle)      │  │
│  │    • Human Approval Gates with Cryptographic Signatures            │  │
│  │    • Typed Multi-Role Handoffs (Inference ≠ Evidence)              │  │
│  └───────────────────────────────────┬────────────────────────────────┘  │
│                                      │                                   │
│            ┌─────────────────────────┴─────────────────────────┐         │
│            ▼                                                   ▼         │
│  ┌───────────────────────────────────┐   ┌────────────────────────────┐  │
│  │ 7. Native In-Process Harness      │   │ 8. Subprocess CLI Adapter  │  │
│  │    Low-latency Go Execution       │   │    Isolated OS Sandbox     │  │
│  └───────────────────────────────────┘   └────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────┘

3. Package Structure

hefesto-lab15-capstone/
├── cmd/
│   └── capstone-cli/
│       └── main.go                  # Entrypoint: 5 operational CLI modes
├── pkg/
│   ├── benchmark/
│   │   └── runner.go                # Equal-compute benchmark runner (SAS vs MAS)
│   ├── budget/
│   │   └── circuit_breaker.go       # 2PC micro-dollar budget governor
│   ├── containment/
│   │   └── dcra_engine.go           # Non-destructive quarantine & audit receipts
│   ├── detector/
│   │   └── inconsistency_rules.go   # O(1) deterministic anomaly detector
│   ├── domain/
│   │   └── types.go                 # Core types, roles, tiers, handoff payloads
│   ├── governance/
│   │   ├── approval_gate.go         # HITL cryptographic approval gates
│   │   └── handoff_validator.go     # Typed role handoffs & evidence segregation
│   ├── harness/
│   │   ├── adapter.go               # AgentHarness interface & registry
│   │   ├── native.go                # In-process Go native harness
│   │   └── subprocess.go            # External CLI subprocess adapter
│   ├── memory/
│   │   └── epistemic_store.go       # Multi-tenant memory store & lineage
│   ├── metrics/
│   │   └── summary.go               # FCE, MTTR, CTax, and certification reporter
│   ├── recovery/
│   │   └── amnesic_agent.go         # Stateless recovery from Merkle checkpoints
│   └── state/
│       ├── fsm.go                   # Concurrent state machine & version clocks
│       └── merkle_ledger.go         # Append-only SHA-256 Merkle chain
├── tests/
│   └── capstone_test.go             # 17/17 automated integration & race tests
├── go.mod                           # Pure standard library (Go 1.26.3, 0 deps)
└── README.md

4. Operational Modes of cmd/capstone-cli

Mode 1: Nominal Execution (-mode nominal)

Executes an end-to-end multi-role task under governed control:

• Planner initiates task, commits planned steps to canonical state.
• 2PC budget governor reserves \$0.005, verifies funds, settles actual consumption (\$0.00342).
• Native harness invokes executor, capturing verified oracle evidence.
• State advances to StateCompleted with 0.0% context contamination.

go run cmd/capstone-cli/main.go -mode nominal

Mode 2: Failure Injection & DCRA Containment (-mode inject-failure)

Simulates an adversarial stale-memory injection attack:

• Initial state: BillingPlan = "free" (at version 1).
• State mutated authoritatively: BillingPlan = "enterprise" (at version 2).
• Adversarial injection: Stale record BillingPlan = "free" resurfaces in retrieval.
• Detector intercepts discrepancy in $\mathcal{O}(1)$.
• Containment revokes operational authority and injects "enterprise" fallback.
• Context contamination: 0.0%. FCE: 100.0%.

go run cmd/capstone-cli/main.go -mode inject-failure

Mode 3: Stateless Recovery (-mode stateless-recovery)

Certifies the *Amnesic Agent Test*:

• Task executes halfway through state machine.
• Process crashes simulated; memory and context are completely wiped.
• New AmnesicAgent spawned with zero conversational context.
• Reconstructs state from SHA-256 Merkle ledger in 120 µs.
• Completes remaining phases to 100% pass without duplicate side-effects.

go run cmd/capstone-cli/main.go -mode stateless-recovery

Mode 4: Equal-Compute Benchmark (-mode benchmark)

Executes SAS vs MAS under an identical \$0.05 compute budget:

• Compares Cost-Per-Success, Tokens-Per-Success, and Coordination Tax.
• Demonstrates SAS superiority for linear logic (CTax = 0.0%).
• Demonstrates MAS superiority for parallel search and security sandboxing.

go run cmd/capstone-cli/main.go -mode benchmark

Mode 5: Industrial Certification Report (-mode certify)

Runs all verification suites and emits an authenticated ASCII + JSON compliance receipt suitable for enterprise governance audits:

go run cmd/capstone-cli/main.go -mode certify

5. Verification Suite

Run all automated unit, concurrency, and race-detection tests:

go test -v -race ./tests

Expected Output:

=== RUN   TestCanonicalStateMonotonicVersions
--- PASS: TestCanonicalStateMonotonicVersions (0.00s)
=== RUN   TestStateMerkleLedgerTamperingDetection
--- PASS: TestStateMerkleLedgerTamperingDetection (0.00s)
=== RUN   TestStateSnapshotIsolation
--- PASS: TestStateSnapshotIsolation (0.00s)
=== RUN   TestMemoryScopedTenancy
--- PASS: TestMemoryScopedTenancy (0.00s)
=== RUN   TestDetectorStaleMemoryTrap
--- PASS: TestDetectorStaleMemoryTrap (0.00s)
=== RUN   TestDetectorAuthorityInversionTrap
--- PASS: TestDetectorAuthorityInversionTrap (0.00s)
=== RUN   TestDetectorCrossScopeBreach
--- PASS: TestDetectorCrossScopeBreach (0.00s)
=== RUN   TestDetectorConflictingMemory
--- PASS: TestDetectorConflictingMemory (0.00s)
=== RUN   TestDCRAQuarantineAuthorityRevocation
--- PASS: TestDCRAQuarantineAuthorityRevocation (0.00s)
=== RUN   TestDCRACanonicalFallback
--- PASS: TestDCRACanonicalFallback (0.00s)
=== RUN   TestBudgetTwoPhaseReservation
--- PASS: TestBudgetTwoPhaseReservation (0.00s)
=== RUN   TestBudgetHardCircuitBreaker
--- PASS: TestBudgetHardCircuitBreaker (0.00s)
=== RUN   TestGovernanceHandoffInferenceSegregation
--- PASS: TestGovernanceHandoffInferenceSegregation (0.00s)
=== RUN   TestGovernanceHumanApprovalGate
--- PASS: TestGovernanceHumanApprovalGate (0.00s)
=== RUN   TestStatelessRecoveryAmnesicAgent
--- PASS: TestStatelessRecoveryAmnesicAgent (0.00s)
=== RUN   TestConcurrentMultiHarnessExecution
--- PASS: TestConcurrentMultiHarnessExecution (0.00s)
=== RUN   TestEqualComputeBenchmarkExecution
--- PASS: TestEqualComputeBenchmarkExecution (0.00s)
PASS
ok      hefesto-lab15-capstone/tests    0.226s

All 17 tests must pass deterministically with zero data races. Congratulations on mastering the architecture of Governed Agent Runtimes!