Hardknock vs LangSmith: Agent Experience vs LLM Tracing
Architectural comparison: LangSmith tracing and evaluation compared with Hardknock's local empirical evidence and controlled counterfactual workflow.
Pre-alpha / roadmap status: The current release supports the local Rust CLI, Git-worktree experiments, SQLite evidence, bounded local chaos campaigns, and the authenticated local Bridge. MCP endpoints, arbitrary agent-requested trials, mandatory pre-tool interception, and privileged or remote sandbox orchestration are roadmap items. Git worktrees provide repository isolation, not a host security sandbox.
Direct Answer & Empirical Comparison Summary
LangSmith provides application observability plus configurable online and offline evaluation, including human, code, and model-based evaluators. Hardknock addresses a different layer: local Git-worktree experiments, evidence capture, and scoped lessons. Hardknock's target integrations could connect those experiments to agent workflows, but mandatory pre-tool mediation remains roadmap work.
Tracing and evaluation help teams inspect and score runs; they are not the same as a controlled counterfactual experiment. LangSmith can use code and human evaluators in addition to model judges. Hardknock's differentiator is the proposed local experiment and evidence workflow, whose agent-native integration is still roadmap-dependent.
Post-hoc telemetry logging & LLM judges vs Active disposable Git Realities & counterfactual trials.
Unstructured JSON span traces vs Typed Lessons, Reflexes, Recoveries, and Experience.
Human review and configured evaluators vs Local evidence and lesson workflow; automatic agent learning is roadmap-dependent.
Hardknock vs LangSmith Tracing & Evaluators: Architectural Matrix
Systematic side-by-side comparison across 8 dimensions of agent experience, counterfactual verification, and workspace safety.
Status icons describe the declared comparison lens for each row; the scope and implementation callout above governs what is currently shipped.
| Evaluation Dimension | Hardknock Experience Engine | LangSmith Tracing & Evaluators |
|---|---|---|
| Primary Substrate | Active empirical experience engine: Git worktree Realities, BLAKE3 environment fingerprints, SQLite store. | Passive distributed tracing: OpenTelemetry-style spans, run trees, prompt-response logs in cloud DB. |
| Causal Verification Method | Paired Counterfactual Trials: Twin Baseline vs Alternative runs under identical environment state. | Configurable evaluators: human review, code rules, heuristics, or model-based judges over traces and datasets. |
| Ontological Rigor | Formal 5-part ontology: Skills (recipes), Lessons (judgments), Reflexes (instincts), Recovery, Experience. | Monolithic trace trees: Spans contain arbitrary key-value inputs, outputs, tokens, and latency metrics. |
| Workspace Impact | Repository-scoped experiments run in disposable Git worktrees and can self-clean; this is not host or network isolation. | Observability records the application run; workspace isolation and cleanup are deployment responsibilities, not a LangSmith requirement. |
| Pre-Execution Intervention | Current scoped retrieval can provide advice; mandatory pre-tool mediation and automatic staging are roadmap items. | LangSmith observes and evaluates application runs; pre-execution authorization is outside its observability role. |
| Cross-Repo Transfer Validation | Can record cross-repository validation evidence when explicitly run; confidence is heuristic and transfer requires revalidation. | Manual dataset curation: Developers manually review trace runs and construct golden benchmark sets. |
| Chaos Engineering & Envelopes | Native Chaos Engine injecting bounded latency, credential expiration, and tool drops to map resilience. | Provides tracing and evaluation primitives; resilience probing requires an evaluation or experiment designed by the user. |
| Deployment & Portability | Local-first open-source CLI (Rust + SQLite) running fully offline or in CI/CD without SaaS lock-in. | LangChain-hosted and self-hosted/hybrid deployment options with configurable trace and evaluation data handling. |
Deep Architectural Breakdown
Technical inspection of Dojo experiment schemas, counterfactual trial definitions, and reflex formation lifecycles.
The Counterfactual vs. Post-Hoc Tracing Paradigm
Why Trace Evaluation and Counterfactual Experiments Answer Different Questions
LangSmith captures trace trees and supports multiple evaluation modes; a model-based judge is one option, not the whole product. A trace can show what happened, while a controlled experiment tests a causal hypothesis under declared conditions. Hardknock's current local substrate supports controlled Git-worktree experiments and evidence records; the agent-requested Dojo workflow shown below is roadmap material unless explicitly implemented.
// Hardknock Dojo: Twin Counterfactual Trial Definition
{
"experiment_id": "exp-dojo-4412",
"base_commit": "e4f9b201a",
"env_fingerprint": "blake3:8f2a1c...",
"hypothesis": "pnpm install succeeds where npm install corrupts lockfile",
"reality_a_baseline": {
"branch": "hardknock/reality-a-4412",
"action": "npm install --save-dev lodash",
"outcome": "FAILURE (Exit 1: ERESOLVE peer dependency collision)"
},
"reality_b_counterfactual": {
"branch": "hardknock/reality-b-4412",
"action": "pnpm add -D lodash",
"outcome": "SUCCESS (Exit 0: lockfile intact, build clean)"
},
"confidence_score": 0.88,
"confidence_note": "Illustrative fixture value; not a calibrated product metric",
"lesson_promoted": "LESSON-PNPM-01"
}Hardknock is intended to improve causal confidence by comparing controlled trials with a declared starting state and variable delta; the result still depends on experiment design and verification checks.
# LangSmith Evaluator (Speculative LLM-as-a-Judge)
from langsmith.evaluation import evaluate
def correctness_evaluator(run, example):
# LLM judges text output without testing code execution
prompt = f"Did the agent correctly fix the migration? Trace: {run.outputs}"
judgment = eval_llm.invoke(prompt) # Subjective guess, no sandbox verification
return {"key": "correctness", "score": 0.5 if "error" in judgment else 1.0}LangSmith uses prompt-based LLM judges to score traces, introducing hallucinated explanations without empirical test execution.
- •Model-based judges can be useful evaluators, but a judge score is not the same as executing and verifying a code change.
- •Hardknock's current local experiments can capture exit codes, stdout/stderr, and filesystem diffs rather than relying only on text evaluation.
- •Controlled counterfactual trials can strengthen causal evidence when the starting state, variable delta, and verification checks are specified.
Ontology & Memory: Skills vs. Lessons vs. Reflexes
How Trace Data Differs from Typed Experience Records
LangSmith trace and dataset records serve observability and evaluation; they are not intended to be a typed agent-memory ontology. Hardknock proposes separate Skills, Lessons, Reflexes, Recovery, and Experience records. In the current implementation, scoped retrieval can provide advice, while mandatory pre-execution interception remains roadmap work.
{
"lesson_id": "les-monorepo-pnpm",
"trigger_markers": ["pnpm-workspace.yaml", "pnpm-lock.yaml"],
"banned_action_signature": "npm install*",
"preferred_action": "pnpm install",
"rationale": "npm install overwrites pnpm lockfile creating phantom workspace collisions",
"evidence_provenance": {
"counterfactual_exp": "exp-dojo-4412",
"verified_on_repos": ["apps/web", "services/api"]
},
"reflex_action": "INTERCEPT_AND_REPLAN",
"validation_tier": "Validated (0.92)"
}Hardknock lessons bind directly to environmental trigger markers and pre-execution reflex actions.
// LangSmith Dataset Row (Unstructured Text)
{
"input": "How do I install dependencies in this repo?",
"output": "Run npm install to install packages.",
"metadata": { "dataset": "onboarding-qa", "created_by": "human_reviewer" }
}A dataset row does not necessarily include environmental triggers, failure signatures, or a pre-execution hook; those fields must be modeled by the surrounding workflow.
- •Flat memory stores do not, by themselves, encode the execution evidence needed to test whether a lesson applies in a particular context.
- •Hardknock can surface scoped advice from recorded lessons; mandatory command interception and workspace mediation remain roadmap capabilities.
- •Provenance tracking is designed to link lessons to the Git state and evidence record used during a trial.
Workspace Safety & Chaos Engineering
Dirty Workspace Pollution vs. Disposable Git Realities & Operating Envelopes
Observability does not determine whether an application's workspace is isolated; that is a deployment choice. Hardknock's current Dojo backend organizes disposable Git worktrees and captures evidence, but those worktrees share the host, network, credentials, and Git objects. Local chaos campaigns are supported within the documented fixture boundary; privileged or remote perturbations and mandatory agent-native mediation remain roadmap work.
{
"chaos_profile": "flaky_infrastructure_envelope",
"perturbations": [
{ "type": "tool_latency_jitter", "min_ms": 500, "max_ms": 4000 },
{ "type": "credential_expiry_mid_flight", "trigger_after_ops": 3 },
{ "type": "partial_tool_drop", "probability": 0.15 }
],
"envelope_boundary_discovered": {
"max_tolerated_latency_ms": 3200,
"recovery_success_rate": "illustrative fixture result",
"result_note": "Not a benchmark; report the harness, sample size, and verification checks"
}
}Illustrative target workflow: bounded local campaigns can record failure observations; production boundaries require representative testing and independent controls.
# Standard agent execution runs directly in the live developer workspace
# Failed experiments leave broken database state and dirty git worktrees
git status
# On branch main
# Changes not staged for commit:
# modified: prisma/schema.prisma (syntax error)
# modified: package-lock.json (corrupted)Standard execution without Dojo isolation pollutes the developer's working tree with half-broken debugging artifacts.
- •Dojo worktrees can reduce contamination of the active checkout during controlled exploration; they are not a host security boundary and require careful cleanup and promotion.
- •Chaos engineering reveals brittle agent assumptions before they cause live system outages.
- •Agents may be evaluated for resilience under controlled stress; learning or production improvement requires an explicit integration and review process.
Illustrative Infrastructure Failure Scenario
Evaluating workspace corruption, cascading failure modes, and recovery reflexes under live engineering conditions.
Monorepo Lockfile Corruption and Cascading Build Failure
An autonomous coding agent is assigned to resolve a critical security vulnerability by updating a shared utility package in a high-traffic TypeScript monorepo.
The agent encounters peer-dependency conflicts during an `npm install` execution. Under standard LLM reasoning, the agent attempts to force resolution with `npm install --legacy-peer-deps`, corrupting the root lockfile and silently desynchronizing 12 workspace packages.
LangSmith records the trace spans and sends the execution logs to the cloud. An LLM-as-a-judge evaluates the run and notes that packages were installed. However, the next CI/CD build fails catastrophically across all 12 services because the lockfile format was corrupted.
Hardknock detects the candidate package mutation in a repository containing `pnpm-workspace.yaml`. The Pre-execution Reflex halts direct execution in the main worktree and stages the task in the Dojo. In Reality A (baseline), `npm install` creates broken lockfile diffs (Exit 1 on build check). In Reality B (counterfactual), `pnpm update` cleanly updates the package and passes all monorepo typechecks. Hardknock promotes the lesson to SQLite, applies the clean fix to main, and discards Reality A.
| Execution Phase | LangSmith Tracing & Evaluators Outcome | Hardknock Empirical Dojo Path |
|---|---|---|
| 1. Task Dispatch | Agent begins editing dependencies directly in the main working tree. Scenario outcome: vulnerable | The documented Dojo workflow can stage the mutation in an isolated Git worktree Reality; mandatory intent interception is roadmap work. Illustrative target outcome: contained [Dojo isolation: zero production git tree pollution.] |
| 2. Failure Reflection | Agent runs npm install --legacy-peer-deps; LangSmith logs trace spans passively. Scenario outcome: vulnerable | Dojo Reality A captures lockfile corruption failure signature; extracts candidate hypothesis. Illustrative target outcome: contained [Failure signature capture: typed stdout/stderr and git diff hashing.] |
| 3. Counterfactual Experiment | No counterfactual testing; LLM judge guesses that installation succeeded. Scenario outcome: vulnerable | Hardknock spawns Reality B under identical BLAKE3 commit snapshot; tests pnpm update. Illustrative target outcome: prevented [Paired counterfactual trial: proving causality across twin branches.] |
| 4. Wisdom Commitment | Dirty, broken lockfile committed to git; downstream CI/CD pipelines break. Scenario outcome: vulnerable | Validated lesson written to SQLite; clean Reality B committed; pre-execution reflex created. Illustrative target outcome: prevented [Lesson/reflex proposal: future package mutations can receive scoped guidance; automatic blocking is not part of the current release.] |
Frequently Asked Questions
Practical questions regarding Dojo worktrees, lesson schemas, reflex arming, and integration with agent frameworks.
Q:Is Hardknock an observability tool like LangSmith or Datadog?
No. Observability tools passively record traces and metrics for human engineers to review on dashboards. Hardknock is an active experience engine designed for the agent itself. It provides disposable execution sandboxes (Dojo Realities), runs paired scientific experiments, and converts empirical failures into durable lessons and automated reflexes.
Q:How does Hardknock run experiments without polluting my Git repository?
Hardknock utilizes Git worktrees (`git worktree add`). It provisions detached, isolated filesystem branches in `.hardknock/dojo/` that mirror the exact current commit. Agents can break code, run tests, and format disk state in these disposable Realities without altering your active working tree or branch history.
Q:What is the difference between an Agent Skill and an Agent Lesson in Hardknock?
A Skill is a procedural recipe for how to perform a task under ideal conditions (the happy path). A Lesson is an empirical judgment that encodes why a task failed, under what environmental conditions, what to do differently, and the counterfactual evidence proving the fix.
Q:How do Hardknock Reflexes prevent repeated agent mistakes?
The current release can record and retrieve scoped lessons keyed to repository markers. A mandatory reflex that intercepts a proposed command (for example, `npm install` in a `pnpm-workspace`) before an external agent executes it is a roadmap capability.
Q:Can Hardknock be used with Claude Code, Cursor, Codex, and custom agent loops?
Hardknock is a lightweight, language-agnostic Rust CLI with SQLite evidence storage, local experiments, and an authenticated local Bridge. The current release does not provide the advertised MCP endpoint or a top-level `hardknock dojo` command; agent-native Dojo commands and mandatory pre-tool mediation are roadmap items.
Give Your Agents Scars in the Dojo
Install Hardknock, run disposable Git Realities, and let your agents fail safely, test counterfactuals, and carry validated lessons forward across codebases.