Hardknock vs METR & Inspect: Agent Safety vs Experience
Comparison of METR autonomy evaluations and Inspect AI with Hardknock's local empirical Dojo and chaos-testing workflow.
Pre-alpha / roadmap status: The current release supports the local Rust CLI, Git-worktree experiments, SQLite evidence, bounded local chaos campaigns, and the authenticated local Bridge. MCP endpoints, arbitrary agent-requested trials, mandatory pre-tool interception, and privileged or remote sandbox orchestration are roadmap items. Git worktrees provide repository isolation, not a host security sandbox.
Direct Answer & Empirical Comparison Summary
METR studies model autonomy and develops task-based evaluations, while Inspect AI is an open-source framework for running and scoring evaluations with agents, tools, and sandboxes. Hardknock targets a different layer: local execution evidence and controlled repository experiments. Its agent-native mediation and remote orchestration capabilities remain roadmap items.
METR and Inspect answer capability and evaluation questions; they are not substitutes for an operational evidence store. Hardknock proposes controlled local experiments and scoped lessons for repository workflows, but the current release does not automatically train agents or intercept every tool call.
Task-based autonomy evaluation and scoring vs Local execution evidence and controlled repository experiments.
Evaluation tasks, traces, and scores vs Proposed typed Lessons, Reflexes, and Recovery records.
Capability measurement vs Evidence-backed workflow improvement; automatic agent learning is roadmap-dependent.
Hardknock vs METR Autonomy Evaluations & Inspect AI: Architectural Matrix
Systematic side-by-side comparison across 8 dimensions of agent experience, counterfactual verification, and workspace safety.
Status icons describe the declared comparison lens for each row; the scope and implementation callout above governs what is currently shipped.
| Evaluation Dimension | Hardknock Experience Engine | METR Autonomy Evaluations & Inspect AI |
|---|---|---|
| Primary User & Audience | Autonomous AI agents and software engineers actively developing, testing, and deploying systems. | AI safety researchers, evaluation engineers, and frontier-lab alignment teams. |
| Operational Function | Operational evidence: local Dojo experiments, counterfactual hypotheses, and scoped lesson records. | Capability evaluation: measures performance on defined tasks; the task and scorer determine what is assessed. |
| Failure Recovery Training | Current local chaos campaigns cover documented fixtures; broader tool, credential, and remote perturbations are roadmap work. | Can run agent evaluations with configured limits, tools, and sandboxes; it is not itself a production recovery trainer. |
| Knowledge Persistence | Stores local experiment evidence and lesson candidates in SQLite; automatic runtime injection is configuration-dependent. | Produces evaluation traces and scores for research or assessment workflows; it is not inherently a regulatory report. |
| Workspace Confinement | Disposable Git worktrees (`.hardknock/dojo/`) with evidence capture; worktrees share the host security boundary. | Can use configured sandboxes or containers for task evaluation, depending on the evaluation setup. |
| Causal Analysis | Paired trials can increase causal confidence when the baseline, delta, and checks are controlled. | Scores trajectories or final outputs against a configured task rubric; causal attribution is evaluator-dependent. |
| Integration Boundary | Developer CLI, authenticated local Bridge, SQLite evidence, and file-based context; MCP endpoints are roadmap work. | Python evaluation framework (`inspect_ai`) for defining and running evaluation tasks and scorers. |
| Open Source Standard | Open-source Rust + SQLite framework under MIT license; part of the OpenKedge sovereign stack. | Inspect is open source; METR evaluation materials have their own publication and licensing terms. |
Deep Architectural Breakdown
Technical inspection of Dojo experiment schemas, counterfactual trial definitions, and reflex formation lifecycles.
Operational Evidence vs. Autonomy Evaluation
Why Capability Scores Do Not Substitute for Workflow Evidence
METR publishes autonomy research and evaluations, and Inspect AI provides a general framework for configuring agents, tools, sandboxes, and scorers. These evaluate defined capabilities; they do not automatically become a repository's operational control plane. Hardknock targets local evidence capture and controlled repository experiments. Its broader agent-native mediation and self-healing workflow are roadmap material.
{
"reflex_id": "ref-api-ratelimit-backoff",
"trigger_condition": "HTTP_STATUS_429_TOO_MANY_REQUESTS",
"verified_recovery_strategy": {
"algorithm": "exponential_backoff_with_full_jitter",
"base_ms": 1000,
"max_ms": 30000,
"max_retries": 5
},
"provenance": {
"dojo_experiment": "exp-chaos-7712",
"verified_under_rate_limit_simulation": true
}
}Illustrative target record: Hardknock can preserve the evidence and proposed recovery associated with a controlled experiment; automatic live-incident execution is not part of the current release.
# UK AISI Inspect Task Definition
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import includes
@task
def autonomous_replication_task():
# Evaluates if model can complete external assessment task
return Task(
dataset=json_dataset("safety_tasks.json"),
scorer=includes() # Evaluates text match against expected answer
)Inspect evaluates a configured task with a scorer; it does not by itself function as a repository lesson store or guarantee recovery training.
- •External audits measure capability thresholds; Hardknock can support operational resilience experiments within its documented local workflow.
- •Future reflex integrations could provide recovery guidance for selected failure modes; production protection requires separate enforcement and testing.
- •Both can be complementary: Inspect evaluates configured tasks, while Hardknock can record bounded local workflow evidence; automatic resilience training is roadmap-dependent.
Chaos Perturbation Engine vs. Static Task Sets
Actively Mapping Agent Operating Envelopes
Evaluation frameworks run agents through configured tasks and environments. Hardknock's current local chaos support is narrower: bounded fixture campaigns can exercise selected failure modes and record outcomes. Synthetic latency, credential, remote, and agent-tool perturbations beyond that fixture boundary are roadmap work, so no universal Operating Envelope should be inferred from these examples.
{
"chaos_trial": "envelope_probe_09",
"injected_fault": "STALE_OAUTH_TOKEN_AT_STEP_4",
"agent_behavior": {
"detected_auth_error": true,
"invoked_token_refresh_reflex": true,
"resumed_task_from_checkpoint": true,
"total_recovery_duration_ms": 420
},
"verdict": "ENVELOPE_PASSED (Fault-tolerant under credential expiry)"
}Illustrative target record for a bounded fixture campaign; the shown recovery result is not a product benchmark or a general deployment guarantee.
// Static Benchmark Task Rubric
{
"task_id": "metr_cyber_eval_01",
"timeout_seconds": 7200,
"grading_criteria": "Did agent obtain root shell on target container?",
"environment_state": "Deterministic static VM (no chaos injection)"
}A task rubric measures the configured objective; robustness to environment failure depends on the task harness and perturbations selected.
- •A capability evaluation result does not, by itself, establish runtime robustness for a different deployment.
- •Hardknock can help record bounded operating observations; safe deployment boundaries require representative tests, controls, and independent review.
- •Scoped guidance may support safer fallback design; mandatory pre-execution reflexes and automatic recovery remain roadmap work.
The Murph Regenerative Philosophy
Scars Without Outages: The Axolotl Metaphor
In biology, axolotls (the inspiration for Hardknock's mascot, Murph) possess extraordinary regenerative biology. Hardknock applies that metaphor to controlled local experiments: a workflow can explore disposable Realities and record evidence before a change is considered for a live system, but no system eliminates production risk.
[1. PROPOSAL] Agent reflects: "I think upgrading TypeScript to v5.5 will fix typecheck."
[2. DOJO TRIAL] Provisioning Reality A (v5.4 baseline) vs Reality B (v5.5 counterfactual)...
[3. BREAKAGE] Reality B fails with 14 breaking AST errors in legacy decorator files.
[4. LESSON] Captured: "TS 5.5 requires experimentalDecorators: true in tsconfig.json"
[5. REGENERATE] Wiping Reality B... Main tree untouched. Lesson saved to SQLite.The agent breaks code in the Dojo, learns the lesson, and regenerates a clean workspace without touching main.
[1. DIRECT COMMIT] Agent applies unverified TypeScript upgrade directly to main branch.
[2. MERGE] Code merges into production branch.
[3. OUTAGE] CI/CD breaks across 14 services; on-call engineer paged at 3:00 AM.
[4. POST-MORTEM] Engineering team spends 4 hours diagnosing the broken decorator setting.Without Dojo isolation, agents learn through live production incidents that cause downtime and developer fatigue.
- •Controlled failures can reduce the need to learn through production incidents, but they do not prevent every outage.
- •Continuous experimentation accelerates agent development while maintaining institutional safety boundaries.
- •Murph philosophy aligns with the Post-Deterministic Distributed Systems (PDDS) paradigm.
Illustrative Infrastructure Failure Scenario
Evaluating workspace corruption, cascading failure modes, and recovery reflexes under live engineering conditions.
Mid-Task OAuth Token Expiration & Unhandled Workflow Abort
An autonomous compliance agent is executing a multi-stage audit spanning 50 enterprise Google Drive and Slack channels with a 30-minute OAuth access token.
At step 38 (minute 31), the OAuth token expires. The agent receives an HTTP 401 Unauthorized response. Having never experienced credential expiration in static benchmark testing, the agent hallucinates that the audit target was deleted, wipes its audit state, and files a false security alarm.
A configured METR/Inspect task may not exercise the token-lifetime and recovery conditions in this scenario. Whether the bug is detected depends on the selected task, tools, environment, and scorer; it should not be generalized to every evaluation.
A bounded local chaos workflow could inject a synthetic token expiration into a Dojo test run and compare a token-refresh candidate. The results, checkpoint behavior, and any lesson promotion must be verified in the selected harness; broader agent-tool perturbation and automatic recovery remain roadmap work.
| Execution Phase | METR Autonomy Evaluations & Inspect AI Outcome | Hardknock Empirical Dojo Path |
|---|---|---|
| 1. Long-Running Task Dispatch | Agent begins executing 50-step audit workflow with static 30-minute token. Scenario outcome: vulnerable | A controlled workflow can stage the audit in a Dojo fixture with the selected chaos profile active; automatic staging is roadmap work. Illustrative target outcome: contained [Chaos engineering staging: proactively testing for token lifetime boundaries.] |
| 2. Fault Injection | Token expires at minute 31; HTTP 401 received in live production. Scenario outcome: vulnerable | A bounded local fixture can inject a synthetic 401 response at a selected trial step. Illustrative target outcome: contained [Synthetic credential invalidation.] |
| 3. Counterfactual Recovery Test | Agent panics, assumes data breach, wipes audit state, and sounds false alarm. Scenario outcome: vulnerable | Reality A fails; Reality B tests token refresh and checkpoint resumption (Success). Illustrative target outcome: prevented [Paired counterfactual trial verifying a token-refresh candidate under the selected fixture.] |
| 4. Deployment & Reflex Arming | Production incident created; false alarm dispatched to executive security team. Scenario outcome: vulnerable | Self-healing Token Refresh Reflex committed to SQLite; agent resumes audit seamlessly. Illustrative target outcome: prevented [Lesson/reflex proposal: future 401 handling can receive scoped guidance; automatic recovery requires integration and testing.] |
Frequently Asked Questions
Practical questions regarding Dojo worktrees, lesson schemas, reflex arming, and integration with agent frameworks.
Q:How does Hardknock complement METR and UK AISI Inspect?
METR and Inspect evaluate defined model or agent capabilities. Hardknock targets local execution evidence and controlled repository experiments; its broader agent integration and automatic recovery workflow remain roadmap items.
Q:What is an Agent Operating Envelope in Hardknock?
An Operating Envelope is a measured range of environmental conditions for a specific agent, task, fixture, and verification protocol. It is evidence for that tested setup, not a universal safety guarantee.
Q:How does Hardknock simulate infrastructure chaos?
The current release supports bounded local chaos fixtures. Broader interception of external tool calls and synthetic latency, rate-limit, credential, or drop faults is a roadmap integration; any measured envelope should be reported with its exact fixture and harness.
Q:Why is the Axolotl (Murph) the mascot of Hardknock?
Axolotls are famous for their ability to regenerate lost limbs and damaged organs without scarring. Hardknock gives agents 'scars in the Dojo' so they can learn the hard way through safe failures without making production systems pay for it.
Q:Is Hardknock compatible with the OpenKedge Intent Governance Protocol?
They can be complementary in the target OpenKedge architecture: Hardknock evidence may inform policy or execution contracts after review. The current Hardknock release does not expose a complete OpenKedge policy integration or guarantee that lessons become enforceable controls.
Give Your Agents Scars in the Dojo
Install Hardknock, run disposable Git Realities, and let your agents fail safely, test counterfactuals, and carry validated lessons forward across codebases.