Build a Self-Harness Workflow

This tutorial is about the workflow before it is about code.

A self-harness loop can be human-led, LLM-assisted, or increasingly automated. The risky part is not that a model proposes a harness change. The risky part is letting that proposal become accepted behavior without replayable evidence.

AIOC is not a self-improvement engine. It is the governance layer underneath a self-harness workflow: the proposal loop can be bold, while validation and promotion stay replayable, comparable, and application-owned.

Teaching example

This tutorial intentionally uses a deliberately aggressive automation model: an LLM-assisted proposal harness reads a problematic RunRecord and proposes both a candidate harness and the expectation used to validate it.

This is not a recommendation to let an AI system update, approve, or promote itself. In a real system, promotion remains application-owned and should match the risk profile of the domain.

Because the proposal harness also drafts the expectation, that expectation is not treated as ground truth. It is treated as part of the proposal and must be reviewed, constrained, or validated according to the application's governance model.

The point of the example is narrower: if an application chooses to automate even part of a self-harness loop, the loop should be grounded in replayable evidence. AIOC provides the validation layer: stored RunRecord values, candidate replay, deterministic comparison, optional judging, and non-regression suites.

This page tells the story in two phases.

The Story

Imagine an application has an agent harness called v1. The harness produces a RunRecord that does not match what the application expects. In this tutorial’s running example, the agent explains photosynthesis, but the answer is not adapted to the learner’s age.

An LLM-assisted workflow reads the problematic RunRecord, diagnoses the issue, and proposes three artifacts:

• a candidate harness, v2;
• a regression suite containing the problematic RunRecord;
• an expectation describing the intended improvement.

AIOC then validates the candidate by replaying the baseline record against v2, comparing baseline and candidate behavior, and optionally asking a judge whether the candidate satisfies the expectation.

If the evidence is acceptable, v2 can be promoted. Otherwise, the proposal is rejected or revised.

Phase 1: First Improvement

Phase 1 creates the first non-regression memory. The application starts with one reported behavior and ends with either an accepted candidate or a rejected proposal.

The important boundary is promotion. AIOC can produce evidence, but it does not decide that v2 should go to production. The application decides whether the evidence is sufficient.

After Phase 1, suite #1 becomes part of the harness history. It is no longer just a test for one old bug. It is a behavioral claim the application chose to preserve.

Phase 2: Non-Regression Memory

Phase 2 is where the workflow becomes more interesting.

Later, the accepted v2 harness produces a different problematic RunRecord. The same LLM-assisted loop can diagnose the new issue and propose a candidate v3, plus a new suite and expectation.

But validation is no longer only about the new issue. AIOC also reruns the suite created in Phase 1. The candidate must fix the new problem without regressing behavior that was previously accepted.

Each accepted harness change becomes part of the future non-regression boundary. The harness can keep changing, but every accepted lesson becomes a test.

Why This Fits AIOC

Self-harness work can sound like it points away from governance. In practice, the more autonomous the proposal loop becomes, the more important governance evidence becomes.

Separate the workflow into three loops:

• proposal loop: human-led, LLM-assisted, or automated;
• validation loop: replayable, comparable, and auditable;
• promotion loop: application-owned.

AIOC belongs in the validation loop. It helps the application answer concrete questions:

• Which RunRecord motivated the change?
• What expectation describes the intended improvement?
• What did the candidate produce when replayed against that record?
• What changed in response, tool calls, policies, guardrails, metadata, and descriptors?
• Did an optional judge agree that the change satisfied the expectation?
• Which previously accepted suites were rerun?

That is the core claim:

AIOC is not a self-improvement engine. It is the governance layer underneath one.

Self-harness proposals can be bold. Promotion should be boring.

Phase 1 Implementation

The runnable Phase 1 example lives in the repository:

examples/self-harness/phase-1.ts

It starts from a stored report:

examples/self-harness/reported-runrecord-1.json

and a baseline harness descriptor:

examples/self-harness/harness-v1.yaml

The JSON report stands in for production persistence, issue intake, or any other place where the application stores the behavior it wants to investigate. In this example, the record contains a photosynthesis question and the original answer. The answer is factually useful, but it is not adapted to an eight-year old learner.

Run the example from the repository root:

OPENAI_API_KEY=... npm run example:self-harness

By default, the example is a dry run. It does not execute the candidate harness. It only asks the proposal harness to draft:

• a diagnosis;
• a candidate v2 harness descriptor;
• a suite name;
• an expectation for the reported case.

The proposal harness receives the smallest useful context:

• the baseline v1 descriptor;
• the reported RunRecord;
• the issue report;
• the AIOC descriptor authoring notes used by this example;
• the allowed capabilities, including get_age_range;
• previous rejection reasons, when the proposal is being retried.

The candidate is not hardcoded. The proposal harness may decide that the best candidate is instruction-only, or it may decide to use the allowed tool. The application does not assume the proposal is correct. It parses the proposed descriptor, checks that it uses only supported capabilities, verifies that tool references are declared correctly, and rejects malformed proposals before any candidate replay happens.

The extra capabilities are intentionally not useful for this issue. They make the choice visible: the proposal harness must decide that get_age_range is the capability that actually explains and fixes the reported behavior.

Rejected proposals are fed back into the next attempt. The example uses one initial proposal plus two retries, and it prints every rejected proposal so the reader can inspect how the loop evolves.

If the static proposal check passes, the default run stops at the dry-run boundary:

=== Dry run boundary ===
Candidate replay is blocked by default. Re-run with --force to execute v2 against RunRecord #1.

This boundary is intentional. Even in a deliberately aggressive automation model, candidate execution is a separate step.

To execute the candidate replay, pass --force:

OPENAI_API_KEY=... npm run example:self-harness -- --force

With --force, the application builds the candidate harness, binds the example tool targets to local tool implementations, and calls runRegressionSuite(...) with the stored RunRecord as the baseline case.

AIOC then:

• replays RunRecord #1 against the candidate harness;
• captures the candidate RunRecord;
• compares baseline and candidate behavior;
• invokes @axiastudio/aioc-regression-judge for the expectation verdict.

The example prints only the final verdict:

=== Final verdict ===
judge: pass
decision: promote v2
summary: The candidate evidence is acceptable: the judge passed and every expected tool was observed.

The judge verdict is evidence, not promotion authority. Promotion remains application-owned. In this example, the application promotes only if the judge passes and the candidate actually used every tool declared by the proposed expectation’s shouldUseTools field.

What Comes Next

Phase 1 creates suite #1: the first accepted behavioral memory. Phase 2 will start from a new problematic RunRecord #2, ask for a v3 candidate, and rerun suite #1 as non-regression memory before accepting the new candidate.

The implementation stays intentionally small. The point is to show how an accepted harness change becomes future non-regression memory, not to build a production-grade self-improvement system.