Hardknock vs SWE-bench: Agent Experience vs Benchmarking
Technical analysis: why SWE-bench static pass@1 benchmarks fail runtime agents, while Hardknock enables live Dojo experimentation and counterfactual lessons.
Pre-alpha / roadmap status: The current release supports the local Rust CLI, Git-worktree experiments, SQLite evidence, bounded local chaos campaigns, and the authenticated local Bridge. MCP endpoints, arbitrary agent-requested trials, mandatory pre-tool interception, and privileged or remote sandbox orchestration are roadmap items. Git worktrees provide repository isolation, not a host security sandbox.
Direct Answer & Empirical Comparison Summary
SWE-bench evaluates configured coding-task performance on benchmark instances. Hardknock targets a separate local workflow for repository-specific experiments and evidence; automatic agent-native Dojo orchestration and lesson promotion are roadmap-dependent.
Static benchmark leaderboards measure retrospective model capability on frozen test suites; they do not by themselves solve an active, novel bug in a proprietary codebase. Hardknock targets a runtime workflow for branching local experiments and recording evidence, subject to the current release and explicit integration.
Offline evaluation harness & static test suites vs Runtime experimentation Dojo & experience engine.
Binary Pass/Fail accuracy metrics vs Dynamic Lessons, Reflexes, and Environment Fingerprints.
Static leaderboard ranking vs In-situ continuous learning and cross-codebase transfer.
Hardknock vs SWE-bench & Static Evaluation Harnesses: Architectural Matrix
Systematic side-by-side comparison across 8 dimensions of agent experience, counterfactual verification, and workspace safety.
Status icons describe the declared comparison lens for each row; the scope and implementation callout above governs what is currently shipped.
| Evaluation Dimension | Hardknock Experience Engine | SWE-bench & Static Evaluation Harnesses |
|---|---|---|
| Primary Objective | Equip agents with an internal empirical learning substrate to experiment, recover, and form reflexes during live coding. | Measure model coding capability across historical GitHub issues to publish comparative leaderboard rankings. |
| Execution Timing | In-situ & Runtime: Invoked by the agent mid-task to branch disposable realities and verify hypotheses. | Post-training & Batch: Invoked offline in massive test sweeps across hundreds of frozen task instances. |
| Knowledge Output | Stores local experiment evidence and lesson candidates in SQLite; automatic pre-execution reflexes and recovery playbooks are roadmap-dependent. | Outputs aggregate statistical scores: Pass@1 rate, resolved percentage, tokens consumed. |
| Adaptation to Novel Codebases | Learns local codebase idiosyncrasies, build tools, and failure patterns through live counterfactual trials. | Fixed evaluation dataset; zero ability to adapt to proprietary internal frameworks or private tools. |
| Hypothesis Testing | Paired counterfactual branching (Reality A vs Reality B) under identical commit states. | Single-shot or multi-turn agent attempt evaluated against a pre-written ground truth unit test patch. |
| Error Boundary Discovery | Chaos Perturbation Engine actively probes latency limits, partial tool drops, and credential timeouts. | Standardized clean environment; does not simulate degraded, flaky, or adversarial production conditions. |
| Workspace Confinement | Detached Git worktrees provide lightweight local repository isolation; creation and cleanup time should be measured on the target filesystem. | Docker container sandboxes per issue; heavy container orchestration requiring large compute clusters. |
| Developer Integration | Developer workflow via the current Rust CLI, local evidence store, and authenticated Bridge; MCP and agent-native Dojo commands are roadmap items. | Evaluation CLI (`swebench.run`) designed for ML research labs running multi-day benchmarking sweeps. |
Deep Architectural Breakdown
Technical inspection of Dojo experiment schemas, counterfactual trial definitions, and reflex formation lifecycles.
The Runtime Experimentation vs. Static Benchmark Paradigm
Why Leaderboards Do Not Produce Resilient Agents
SWE-bench evaluates whether an agent can resolve a frozen historical issue under the configured benchmark harness and reports per-instance outcomes plus aggregate metrics. It is not intended to provide a live repository lesson workflow during unrelated software development. Hardknock targets that separate operational layer with local worktree experiments and evidence capture; agent-native Dojo orchestration and automatic promotion remain roadmap material.
# Agent invokes Hardknock Dojo mid-task to test alternative refactoring strategies
hardknock experiment create --name resolve-database-deadlock
# Illustrative workflow: exact subcommands depend on the active CLI release.
hardknock experiment run --name resolve-database-deadlock --command "npm test -- test/concurrency.test.ts"
hardknock why --lesson LESSON-DATABASE-DEADLOCKIllustrative target workflow: the current CLI supports local experiments and evidence capture, while this agent-native orchestration is roadmap material.
# SWE-bench offline batch evaluation script
python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --predictions_path gold_predictions.jsonl --max_workers 8 --run_id evaluation_run_01SWE-bench runs static batch evaluation across historical issues to generate research leaderboard statistics.
- •Static benchmarks measure configured task performance; they do not automatically control a separate runtime workspace.
- •Hardknock can record repository-specific evidence through an explicit local workflow; automatic agent learning remains roadmap-dependent.
- •Live counterfactual trials can make a hypothesis more testable than sequential guessing when the experiment is properly controlled.
Ontology: Scalar Pass@1 vs. Durable Empirical Lessons
How Hardknock Preserves Operational Wisdom
SWE-bench produces per-instance evaluation outcomes and aggregate metrics for a fixed benchmark workflow; it is not designed as a repository-specific lesson store. Hardknock can capture failure signatures and experiment evidence in SQLite through an explicit workflow. Reuse in another repository requires matching triggers and revalidation rather than assuming automatic transfer.
{
"lesson_id": "les-db-deadlock-04",
"domain": "postgres-concurrency",
"failure_signature": "ERROR: deadlock detected (SQLSTATE 40P01)",
"tested_strategy": "SELECT ... FOR UPDATE with explicit transaction ordering",
"empirical_proof": {
"trials_passed": "illustrative",
"trials_failed": "illustrative",
"concurrency_level": "declared by the test harness",
"result_note": "Example record; not a benchmark"
},
"confidence": "heuristic_example",
"confidence_note": "Illustrative record; not a calibrated benchmark",
"applies_to_files": ["*.repository.ts", "prisma/schema.prisma"]
}Hardknock records structured failure signatures and empirical proofs that persist across coding sessions.
// SWE-bench Raw Instance Result
{
"instance_id": "django__django-11099",
"model_patch": "diff --git a/django/db/models... +++ b/django/db/models...",
"resolved": false,
"test_output": "FAIL: test_deadlock (db.tests.TransactionTests)"
}SWE-bench produces binary pass/fail results that are discarded after the benchmark evaluation finishes.
- •SWE-bench results do not feed back into agent memory or operational reflexes.
- •Hardknock can record selected failure evidence as lesson candidates; retention and delivery depend on the configured workflow.
- •Lessons can be shared across an entire enterprise fleet of autonomous coding agents.
Operational Envelope Mapping vs. Clean-Room Testing
Simulating Chaos, Latency, and Partial Tool Failures
SWE-bench uses a reproducible harness for its benchmark tasks, but exact task environments and checks are configuration-dependent; it is not intended to be a continuous production-chaos system. Hardknock's current chaos support is bounded by local fixtures, while broader latency, credential, tool, and agent-recovery perturbations are roadmap work.
{
"agent_id": "claude-code-dev-4",
"task": "distributed_database_migration",
"operating_envelope": {
"safe_latency_range_ms": [0, 2800],
"flaky_tool_resilience_score": 0.91,
"token_rotation_recovery_verified": true
},
"identified_failure_mode": "Agent enters infinite retry loop if API returns HTTP 429 without Retry-After header",
"remedy_reflex_compiled": "REFLEX-BACKOFF-JITTER-429"
}Hardknock discovers an agent's failure boundaries under synthetic stress and compiles automated recovery reflexes.
# SWE-bench sterile Docker container
FROM python:3.9-slim
# Clean, deterministic, non-flaky test environment
RUN apt-get update && apt-get install -y git build-essential
# Zero network perturbations, zero credential expirations, zero latencySWE-bench runs in clean-room containers that fail to test whether agents can handle real-world infrastructure chaos.
- •Clean-room benchmarks give a false sense of security regarding agent production readiness.
- •Hardknock's chaos engine proactively identifies fragile failure modes before agents touch live systems.
- •A future reflex workflow could support recovery guidance when the selected fixture degrades.
Illustrative Infrastructure Failure Scenario
Evaluating workspace corruption, cascading failure modes, and recovery reflexes under live engineering conditions.
Concurrent Database Deadlock Remediation in High-Throughput Service
An autonomous backend engineering agent is assigned to fix a high-priority database deadlock issue causing 500 errors in a financial transaction service.
The agent attempts to fix the deadlock by wrapping queries in optimistic retry blocks. Under high concurrency (100 simultaneous transactions), the optimistic retries cause thread exhaustion and crash the database connection pool.
SWE-bench-style static testing passes because the single-threaded unit test in the test suite runs with concurrency=1. The agent deploys the change, causing a complete database crash in staging under live load.
Illustrative target workflow: stage the refactoring in a Dojo and compare twin Realities under a declared concurrency fixture. Reality A tests optimistic retries, while Reality B tests pessimistic locking with canonical key ordering. The selected checks could reject the failing variant and record evidence for the passing variant; automatic lesson registration and promotion depend on the implementation.
| Execution Phase | SWE-bench & Static Evaluation Harnesses Outcome | Hardknock Empirical Dojo Path |
|---|---|---|
| 1. Problem Ingestion | Agent reads deadlock error log and writes optimistic retry logic in main branch. Scenario outcome: vulnerable | Hardknock provisions isolated Dojo Realities; stages concurrency experiments. Illustrative target outcome: contained [Dojo worktree isolation: testing complex concurrency without live database risk.] |
| 2. Single-Threaded Test | Agent runs local unit test; test passes with concurrency=1. Scenario outcome: vulnerable | Hardknock activates Chaos Engine; spins up concurrent synthetic load in Reality A and B. Illustrative target outcome: contained [Chaos perturbation: subjecting candidate fixes to realistic operational concurrency.] |
| 3. Counterfactual Stress Test | No stress test performed; agent assumes fix is complete. Scenario outcome: vulnerable | Illustrative Reality A/B result: the candidate strategy passes the selected concurrency checks while the baseline fails; exact rates require a measured harness. Illustrative target outcome: prevented [Paired counterfactual proof under load.] |
| 4. Deployment & Reflex Update | Flawed code deployed to staging; connection pool exhausted. Scenario outcome: vulnerable | Verified Reality B merged to main; Lesson and Concurrency Reflex stored in SQLite. Illustrative target outcome: prevented [Empirical wisdom committed for future concurrency refactorings.] |
Frequently Asked Questions
Practical questions regarding Dojo worktrees, lesson schemas, reflex arming, and integration with agent frameworks.
Q:Can I use SWE-bench and Hardknock together?
Yes. SWE-bench is an excellent benchmarking harness for measuring how your underlying foundation models score on historical benchmarks. Hardknock is the empirical experience engine you deploy inside your agent toolchain so it can experiment safely and learn while working on active repositories.
Q:How does Hardknock capture failure signatures without human intervention?
Hardknock hooks into command execution streams, capturing stdout, stderr, process exit codes, affected file diffs, and environment fingerprints. It hashes these attributes into a unique failure signature and prompts the agent to formulate candidate explanations.
Q:Why are static pass@1 benchmarks misleading for autonomous agents?
Pass@1 benchmarks evaluate whether an agent can solve a problem on its first attempt in a clean, isolated container. In reality, autonomous agents must iterate, debug errors, recover from failed tool calls, and adapt to repo-specific nuances—capabilities that static benchmarks do not measure.
Q:Does Hardknock require Docker containers for every experiment?
No. The current release uses Git worktrees for lightweight local repository isolation. Worktrees share the host kernel, network, credentials, and Git objects, so they are not equivalent to Docker, E2B, or gVisor sandboxes; remote/container orchestration is roadmap work.
Q:How does Hardknock validate that a lesson transfers to other codebases?
Hardknock can attach heuristic confidence and provenance to lessons. Cross-repository reuse should be treated as a new validation obligation; the current materials do not establish calibrated 0.78/0.90 thresholds or universal transfer.
Give Your Agents Scars in the Dojo
Install Hardknock, run disposable Git Realities, and let your agents fail safely, test counterfactuals, and carry validated lessons forward across codebases.