Hardknock vs Mem0 & Letta: Agent Experience vs Memory
In-depth comparison: memory systems such as Mem0 and Letta provide persistent context, while Hardknock focuses on execution evidence and controlled experiments.
Pre-alpha / roadmap status: The current release supports the local Rust CLI, Git-worktree experiments, SQLite evidence, bounded local chaos campaigns, and the authenticated local Bridge. MCP endpoints, arbitrary agent-requested trials, mandatory pre-tool interception, and privileged or remote sandbox orchestration are roadmap items. Git worktrees provide repository isolation, not a host security sandbox.
Direct Answer & Empirical Comparison Summary
Mem0 provides persistent memory with vector, metadata, and graph-oriented capabilities; Letta provides stateful agents that can manage memory over time. Neither product is primarily a Git-worktree experiment runner. Hardknock focuses on execution evidence, controlled trials, and scoped lessons, so the systems are complementary rather than interchangeable.
Persistent memory can preserve useful context, but any remembered claim still needs appropriate validation for the current environment. Hardknock adds execution evidence and controlled trials for claims that require empirical verification; it does not replace general-purpose memory or guarantee universal correctness.
Vector embeddings of conversation text vs SQLite store of immutable Git snapshots & counterfactual proofs.
Unstructured natural language facts vs Typed Skills, Lessons, Reflexes, and Failure Signatures.
Top-k semantic retrieval vs Trigger-marker scoped injection and empirical invalidation.
Hardknock vs Mem0 & Letta (MemGPT) Memory Frameworks: Architectural Matrix
Systematic side-by-side comparison across 8 dimensions of agent experience, counterfactual verification, and workspace safety.
Status icons describe the declared comparison lens for each row; the scope and implementation callout above governs what is currently shipped.
| Evaluation Dimension | Hardknock Experience Engine | Mem0 & Letta (MemGPT) Memory Frameworks |
|---|---|---|
| Data Representation | Empirical Ground Truth: Git commit SHA, environment BLAKE3 digest, command exit codes, paired trial diffs. | Persistent memory records that may include extracted facts, preferences, summaries, metadata, vectors, and graph relationships; validation is application-dependent. |
| Belief Invalidation | Counterfactual refutation can support confidence review when a verification check fails; automatic downgrade is integration-dependent. | Memory systems provide persistence and retrieval; invalidation and revalidation behavior depends on the configured product and application workflow. |
| Ontological Structure | Strict 5-part ontology: Skills, Lessons, Reflexes, Recoveries, and Experience. | Memory-oriented data models, including structured and graph-backed representations, rather than Hardknock's experiment/evidence ontology. |
| Retrieval Precision | Deterministic Trigger Matching: Injected only when environmental markers (e.g., `pnpm-workspace.yaml`) exist. | Probabilistic Vector Similarity: Retrieves memories based on cosine similarity of user prompts, risking false matches. |
| Hallucination Resistance | Evidence-Backed: Every stored lesson is linked to the exact counterfactual trial where it was proven. | Retrieved memories are context for the agent; their correctness depends on how they were created, scoped, and validated. |
| Execution Sandboxing | Native Dojo with disposable Git worktree Realities to test memories before applying them to production. | Memory frameworks focus on persistence and retrieval; execution isolation is supplied by the surrounding agent and deployment. |
| Pre-Execution Defense | Current scoped retrieval can provide advice; mandatory pre-tool interception is a roadmap item. | Memory retrieval supplies context; whether a tool call is blocked or approved is an agent/deployment responsibility. |
| Storage & Query Substrate | Local SQLite with typed relational tables, foreign key constraints, and JSON schemas (offline-first). | Product-specific storage, which may combine relational, vector, graph, and metadata stores depending on configuration. |
Deep Architectural Breakdown
Technical inspection of Dojo experiment schemas, counterfactual trial definitions, and reflex formation lifecycles.
The Empirical Ground Truth vs. Semantic Echo Chamber Paradigm
Execution Evidence vs. Persistent Agent Memory
Mem0 and Letta are designed to preserve agent context and memory across interactions. Those records answer a different question from an execution experiment: what context should be recalled versus what happened under a declared environment and check. Hardknock's current evidence model is intended for the latter; any lesson remains scoped to the tested conditions and should be revalidated when those conditions change.
// Hardknock Experience Record in SQLite
{
"experience_id": "exp-sql-8821",
"provenance": {
"git_commit": "c812d44",
"env_fingerprint": "blake3:99a8f102...",
"os": "linux-x86_64",
"runtime_versions": { "node": "v22.1.0", "pnpm": "9.1.0" }
},
"action_executed": "pnpm build --filter=@core/db",
"exit_code": 0,
"stdout_digest": "sha256:3b19fc...",
"counterfactual_baseline_id": "exp-sql-8820", // The failed attempt
"verified_causal_delta": "Added export * from './schema' to index.ts"
}Hardknock records experiment provenance such as environment digests, exit codes, and diffs when available.
// Mem0 Vector Memory Record
{
"id": "mem-text-102",
"memory": "In this project, always export database models using default export.",
"user_id": "agent-user",
"score": 0.89 // Cosine similarity score, unverified against actual codebase
}Mem0 stores unverified text assertions that may be technically incorrect or hallucinated by previous model turns.
- •Vector memory does not inherently distinguish proven facts from plausible hallucinations.
- •Hardknock allows agents to verify claims empirically by replaying experiments against historical snapshots.
- •Recorded causal diffs can provide evidence for a lesson, but promotion still requires a defined verification workflow.
Deterministic Trigger Matching vs. Fuzzy Semantic Search
Eliminating Irrelevant Context Pollution
Semantic vector search retrieves memory chunks by embedding similarity, which can require additional application-level filtering. Hardknock proposes concrete trigger markers (for example, file existence, package dependencies, and error patterns) to narrow retrieval. The current workflow can format matching lessons, but the agent must be configured to consume them and revalidation remains necessary.
{
"lesson_id": "les-rust-wasm-opt",
"triggers": {
"file_markers_required": ["Cargo.toml", "wasm-pack.config.js"],
"action_matcher": "wasm-pack build*",
"env_constraint": "arch == x86_64"
},
"injected_guidance": "wasm-pack build requires --release flag to avoid binary size bloat in CI",
"priority": "HIGH"
}Illustrative target: lessons can be narrowed by declared filesystem markers instead of relying only on embedding similarity.
# Letta / MemGPT Vector Query
results = memory.search(
query="build WebAssembly bundle",
top_k=3 # Retrieves memories based on semantic similarity of text embeddings
)Fuzzy semantic vector search frequently retrieves obsolete or out-of-context memory chunks.
- •Fuzzy vector retrieval pollutes LLM context windows with out-of-context memories, increasing inference costs.
- •Hardknock trigger markers narrow retrieval to declared environmental conditions; they do not establish 100% semantic correctness or replace revalidation.
- •Context injection is scoped to `.hardknock/context.md` for immediate consumption by coding agents.
Belief Invalidation: Re-Testing Stale Assumptions
How Hardknock Detects and Prunes Obsolete Lessons
Codebases evolve: libraries are upgraded, APIs change, and old workarounds become obsolete. A memory system may retain stale material unless the application adds freshness and validation rules. Hardknock proposes an invalidation loop: when a verification check fails, the workflow can lower confidence, flag a lesson for re-evaluation, and schedule a controlled trial. The current release does not guarantee that this happens automatically for every agent action.
{
"lesson_id": "les-legacy-webpack-patch",
"status": "INVALIDATED",
"invalidation_event": {
"detected_on_commit": "f109b82",
"reason": "Project upgraded to Vite; webpack patch is no longer applicable",
"trial_outcome": "Failure in Reality B (Vite config ignores webpack rules)",
"confidence_updated": "heuristic_example",
"confidence_note": "Illustrative value; not a calibrated score"
}
}Target lifecycle example: a controlled workflow can downgrade or invalidate a stale lesson when an environmental test fails.
// Vector DB continues returning deprecated memory forever:
// "Remember: Always add webpack-node-externals to webpack.config.js"
// The agent attempts to apply this to a new Vite project, causing build failures.Vector databases have no mechanism to test or invalidate stale memories against active codebases.
- •Static vector memories become technical debt that degrades agent performance over time.
- •A proposed invalidation lifecycle can prioritize empirically supported rules, but it does not ensure that agents act only on current rules without integration and revalidation.
- •Confidence scoring can record a heuristic validity signal; ongoing validity still requires revalidation.
Illustrative Infrastructure Failure Scenario
Evaluating workspace corruption, cascading failure modes, and recovery reflexes under live engineering conditions.
Hallucinated Package Import Vector Memory Poisoning
An enterprise autonomous agent is tasked with writing serverless API endpoints using an internal proprietary SDK (`@corp/core-auth`).
In a previous conversation turn, an LLM hallucinated that `@corp/core-auth` contained a method called `verifyUserSessionSync()`. The vector memory framework stored this text chunk. When a new agent session begins, the memory framework retrieves the hallucinated fact, causing the agent to write broken code across 8 microservices.
Mem0 / Letta retrieves the memory chunk: 'Use verifyUserSessionSync() for authentication'. The agent uses the non-existent function, failing typechecks and breaking build pipelines repeatedly.
In the target workflow, the Dojo would test the claim before promotion. In this illustrative example, Reality A reports TS2339 for `verifyUserSessionSync()` and a candidate async method succeeds; promotion and SQLite persistence require the implemented validation workflow.
| Execution Phase | Mem0 & Letta (MemGPT) Memory Frameworks Outcome | Hardknock Empirical Dojo Path |
|---|---|---|
| 1. Memory Retrieval | Vector memory injects hallucinated text: 'Use verifyUserSessionSync()'. Scenario outcome: vulnerable | Hardknock checks SQLite for verified lessons; finds no empirical proof for sync method. Illustrative target outcome: contained [Target evidence rule: retrieve lessons with recorded provenance rather than treating arbitrary text as verified.] |
| 2. Action Formulation | Agent writes code calling non-existent sync function in live workspace. Scenario outcome: vulnerable | Agent proposes candidate implementation; Hardknock stages code in Dojo Reality. Illustrative target outcome: contained [Dojo staging: isolating untried code before committing to main.] |
| 3. Compilation & Verification | Main repo build fails with TypeScript compiler error. Scenario outcome: vulnerable | Dojo compilation fails in Reality A; agent reflects and tests async method in Reality B (Success). Illustrative target outcome: prevented [Counterfactual verification: discovering the correct API through empirical testing.] |
| 4. Memory Commitment | Agent repeats hallucination; vector store retains corrupted memory. Scenario outcome: vulnerable | Verified lesson for async authentication committed to SQLite with compiler proof. Illustrative target outcome: prevented [Evidence persistence: retain API-usage records with explicit provenance and validation status.] |
Frequently Asked Questions
Practical questions regarding Dojo worktrees, lesson schemas, reflex arming, and integration with agent frameworks.
Q:How does Hardknock differ from vector memory frameworks like Mem0 and Letta?
Mem0 and Letta store unstructured text chunks and conversation summaries in vector databases using semantic embeddings. Hardknock is an empirical experience engine that stores verifiable execution records: git commit snapshots, environment fingerprints, stdout/stderr diffs, and paired counterfactual trial proofs in SQLite.
Q:Why does semantic vector search cause problems for coding agents?
Vector search relies on embedding similarity and may retrieve out-of-context material without additional filters. It does not inherently prove whether a remembered assertion is true; validation and freshness controls remain application responsibilities.
Q:How does Hardknock inject experience into coding agents?
Hardknock can format matching lessons as structured Markdown for `.hardknock/context.md`. An agent receives that context only if its runtime is configured to read the file; automatic ingestion is not universal.
Q:What happens when an environment changes and an old lesson is no longer valid?
When a verification check fails, a controlled workflow can downgrade a lesson, flag it for re-evaluation, and stage a follow-up experiment. Automatic triggering depends on the integration and is not guaranteed by the current release.
Q:Is Hardknock local or cloud-based?
Hardknock is local-first and open-source. It runs entirely on your local machine or CI/CD runner using a Rust binary and a local SQLite database (`.hardknock/store.db`), requiring zero external cloud dependencies or API keys.
Give Your Agents Scars in the Dojo
Install Hardknock, run disposable Git Realities, and let your agents fail safely, test counterfactuals, and carry validated lessons forward across codebases.