Lab L12 — Trajectory Tracing & Decision Lineage Engine (Pure Go)

Phase 9 · OBSERVABILITY & EVIDENCE — Lab L12
Status: Authored & Empirically Verified.
Student Lab Package: Authenticated Direct Download from VTAlgo Platform (hefesto-lab12-trajectory-lineage.zip)
Branches: main (starter template) · solution (reference architecture).
Canonical Path: Student repo only (hefesto-lab12-trajectory-lineage) — not E:\bridle, not the live product.

1. Laboratory Objective

Construct from scratch in pure Go standard library an enterprise-grade Trajectory Tracing & Decision Lineage Engine, proving empirically that autonomous multi-agent workflows can be made fully observable, mathematically auditable, and deterministically reproducible with zero runtime performance penalty.

The student implements:

• An atomic trajectory engine (pkg/trajectory) organizing execution into immutable RunID runs, strictly sequenced Turn instances, hierarchical OpenTelemetry-compatible Span trees, and an acyclic causal DAG with cycle detection.
• A decision lineage engine (pkg/lineage) encoding the cardinal formulation: $$\text{STATE} + \text{EVIDENCE} + \text{ADMITTED CONTEXT} + \text{RETRIEVED MEMORY} \longrightarrow \text{DECISION} \longrightarrow \text{ARTIFACT / ACTION}$$ supported by four strict authority tiers (State, Evidence, Context, Memory) and cryptographic SHA-256 sealing for mathematical non-repudiation.
• A zero-trust memory observability auditor (pkg/memoryobs) capturing all six lifecycle events (retrieved, admitted, rejected, demoted, contradiction_detected, scope_violation), enforcing minimal provenance profiles, and recording structured, data-minimized rejection logs.
• A content-addressed artifact provenance graph (pkg/artifact) linking generated files to execution turns and deterministic oracle verification receipts via SHA-256 digests.
• A micro-financial telemetry monitor (pkg/telemetry) tracking microsecond execution durations, prompt/completion token consumption, and dynamic USD cost attribution against rate cards, equipped with hard budget circuit breakers.
• A unified command-line utility (cmd/lineage-cli) supporting nominal execution, memory rejection auditing, and deterministic forensic post-mortem replay.
• A 24/24 deterministic test suite passing in $<250$ ms with zero external third-party dependencies in go.mod.

2. System Architecture

┌──────────────────────────────────────────────────────────────────────────┐
│        HEFESTO TRAJECTORY & DECISION LINEAGE ENGINE (PURE GO)            │
│                                                                          │
│  ┌─────────────────────────┐           ┌──────────────────────────────┐  │
│  │ 1. Causal Trajectory    │           │ 2. Decision Lineage          │  │
│  │    RunID · Turns (0..N) ├──────────►│    Cardinal Equation:        │  │
│  │    Hierarchical Spans   │           │    S + E + C + M ──> DECISION│  │
│  └───────────┬─────────────┘           └──────────────┬───────────────┘  │
│              │                                        │                  │
│              ▼                                        ▼                  │
│  ┌─────────────────────────┐           ┌──────────────────────────────┐  │
│  │ 3. Memory Observability │           │ 4. Artifact Provenance Graph │  │
│  │    6 Lifecycle Events   ├──────────►│    SHA-256 Content Addressing│  │
│  │    Rejected Audit Logs  │           │    Oracle Verification Sealed│  │
│  └───────────┬─────────────┘           └──────────────┬───────────────┘  │
│              │                                        │                  │
│              ▼                                        ▼                  │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │ 5. Telemetry & CLI: Tokens · USD Spend · Forensic Replay in <250ms │  │
│  └────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────┘

3. The Five Core Engine Subsystems

1. Causal Trajectory Engine (pkg/trajectory)

• Turn Specification: Encapsulates Index, Timestamp, AgentID, ModelID, PriorState, Observation, ActionProposal, PolicyDecision, and ResultingState.
• Span Model: Compatible with OpenTelemetry GenAI standards (TraceID, SpanID, ParentSpanID, Attributes, Status).
• VerifyCausalAcyclicity(): Detects recursive parent cycles across distributed agent invocations in $O(V+E)$ time.

2. Decision Lineage Engine (pkg/lineage)

• Authority Tiers:
• TierState (1): Canonical ground-truth state (commits, active branches).
• TierEvidence (2): Empirical evidence (compiler outputs, $100\%$ test receipts).
• TierContext (3): Admitted user context and system instructions.
• TierMemory (4): Historical memory records (tentative suggestions).
• VerifyAuthorityGenealogy(decisionID): Re-evaluates SHA-256 hashes of all input nodes, checks top-level lineage digest consistency, and asserts that historical memory cannot override canonical state without explicit governance enforcement.

3. Memory Observability & Rejection Auditor (pkg/memoryobs)

• Lifecycle Events: memory.retrieved, memory.admitted, memory.rejected, memory.demoted, memory.contradiction_detected, memory.scope_violation.
• Minimal Provenance Profile: Embeds MemoryRef (SHA-256), OriginatingSource, OriginatingRole, ScopeTenantID, ScopeTaskID, RecordAge, AuthorityClass, GovernanceOutcome, and RuleCitation.
• Data-Minimized Audit Log: Persists structured metadata of rejected and blocked candidates without storing gigabytes of redundant raw context.

4. Content-Addressed Artifact Provenance (pkg/artifact)

• ArtifactNode: Binds file path to SHA-256 content digest, file size, originating turn, and deterministic verification receipt.
• VerifyArtifactIntegrity(path, content): Detects out-of-band file tampering instantly in constant time.

5. Telemetry & Financial Attribution (pkg/telemetry)

• Rate Cards: Configurable input and completion pricing per $1,000,000$ tokens in USD (e.g. Claude 3.7 Sonnet: $\$3.00$ / $\$15.00$).
• Microsecond Latency: Nanosecond wall-clock tracking mapped to microsecond durations.
• Circuit Breaker: Deterministically interrupts execution with ErrCostBudgetExceeded or ErrTokenBudgetExceeded whenever assigned task quotas are breached.

4. CLI Multi-Mode Operations

The student CLI (cmd/lineage-cli) exposes three deterministic operational modes:

Mode 1: Nominal Trajectory Execution

go run ./cmd/lineage-cli -mode nominal

Executes a multi-turn workflow, creates artifacts, records decisions under the cardinal equation, verifies authority genealogy, and displays token and USD spend.

Mode 2: Memory Rejection & Scope Audit

go run ./cmd/lineage-cli -mode audit-rejections

Evaluates memory candidates, trapping cross-tenant violations, stale TTL records, and state contradictions, emitting a structured JSON rejection audit trail.

Mode 3: Deterministic Forensic Post-Mortem Replay

go run ./cmd/lineage-cli -mode forensic-replay

Replays a recorded incident trajectory step-by-step in an isolated sandbox with zero LLM API calls, verifying root-cause analysis (RCA) security assertions.

5. Verification & Test Suite

Run the full automated test suite:

go test -v ./...

Test Suite Inventory (24/24 PASS in <250ms)

• tests/trajectory_test.go:
• TestTrajectory_TurnSequencing: Monotonic sequential turn enforcement.
• TestTrajectory_SpanAcyclicity: Causal DAG cycle detection.
• TestTrajectory_JSONSerialization: Round-trip OTLP JSON serialization.
• TestTrajectory_ConcurrentAccess: Thread-safe RWMutex operations.
• tests/lineage_test.go:
• TestLineage_CardinalEquation: 4-tier authority verification.
• TestLineage_TamperedNodeDetected: Input tampering detection.
• TestLineage_TamperedLineageDigest: Top-level seal tampering detection.
• TestLineage_MemoryOverrideViolation: Precedence T1 > T4 enforcement.
• TestLineage_ConcurrentDecisions: Concurrent decision recording.
• tests/memoryobs_test.go:
• TestMemoryObs_AdmissionAndRejection: All 6 lifecycle events tested.
• TestMemoryObs_RejectedAuditLog: Data-minimized rejection log verification.
• TestMemoryObs_ConcurrentAuditing: Concurrent candidate evaluation.
• tests/artifact_test.go:
• TestArtifact_RegisterAndVerify: SHA-256 node registration.
• TestArtifact_TamperedContentFails: Tampered content failure.
• TestArtifact_GraphEdges: Causal edge linking.
• TestArtifact_ConcurrentAccess: Concurrent artifact registration.
• tests/telemetry_test.go:
• TestTelemetry_CostAndTokenCalculation: Exact USD cost attribution.
• TestTelemetry_TokenBudgetExceeded: Hard token limit circuit breaking.
• TestTelemetry_CostBudgetExceeded: Hard dollar limit circuit breaking.
• TestTelemetry_OTelAttributes: OpenTelemetry GenAI conventions.
• TestTelemetry_ConcurrentRecording: Concurrent telemetry updates.
• tests/cli_test.go:
• TestCLI_NominalWorkflow: End-to-end nominal pipeline.
• TestCLI_AuditRejectionsWorkflow: Rejections and scope traps.
• TestCLI_ForensicReplayWorkflow: Deterministic replay and RCA assertions.