If you're building AI systems — agents, RAG pipelines, fine-tuned models, anything that learns — you already know that unit tests are necessary and completely insufficient. Tests tell you whether a function returns the right value. They can't tell you whether your system understands what it learned. Whether your orchestrator proposes sensible work. Whether knowledge transfers meaningfully between agents.
These are system-level questions. Assert statements don't reach them.
Most teams handle this informally. Run the system, look at the data, find something that looks good, call it a win. The research community has a name for this: HARKing — Hypothesizing After Results are Known. You find something promising and retroactively frame it as what you were testing all along. Nobody does it deliberately. But without targets committed to version control before data collection, there's no way to prove you weren't.
That's not science. That's storytelling with numbers.
What HDD Actually Is
Hypothesis-Driven Development wires the scientific method into the engineering workflow itself — pre-registered hypotheses in version control, quantitative targets committed before data collection, and a branch-per-experiment model that only merges validated results. The foundations aren't new: Eric Ries introduced build-measure-learn loops in 2011, Jeff Gothelf and Josh Seiden formalized hypothesis-driven product development in Lean UX, and pre-registration has been standard in clinical research since the mid-2000s. What HDD adds is the machinery to make these practices automatic and auditable in a software development context.
The cycle:
- Capture the question. "Would entity linking help agents understand what they're reading earlier in training?"
- Formalize a hypothesis. "Entity linking increases knowledge graph density by at least 15%." Specific. Quantitative. Stated in advance.
- Design and run the experiment. Train one agent with it, one without. Compare.
- Accept the result. Hit the target, or didn't.
The discipline lives in step 2. The hypothesis and its threshold are committed to version control before any data is collected. You can't move the goalposts. You said 15%. Either you hit it or you didn't.
How It Differs from TDD
| TDD | BDD | HDD | |
|---|---|---|---|
| Unit of work | Function | Behavior | Hypothesis |
| What you write first | A failing test | A scenario spec | A falsifiable claim |
| Pass criteria | Assert equals | Gherkin steps | Quantitative target |
| What a failure means | Bug | Regression | Valid science |
| Discovery phase | None | Informal | Literature review |
Two differences matter. First: HDD starts with a question, not a test. Before formalizing a hypothesis, you do a literature review — often LLM-assisted — to understand what prior work exists and what targets are realistic. TDD assumes you know what to build. HDD assumes you need to figure out what's even worth building.
Second: a refuted hypothesis isn't a bug. It's data. In TDD, red means "fix this." In HDD, red means "now we know something we didn't know before."
The Story: When Failure Is the Contribution
Here's the example I keep coming back to. We were building predictive processing — can an AI agent anticipate what the user will ask next? We formulated five hypotheses with specific targets and ran them against a live agent with 66,000+ knowledge triples.
| Hypothesis | Target | Actual | Result |
|---|---|---|---|
| Meaningful predictions | >=60% accuracy | 54% | Refuted |
| Improves over sessions | +15pp over 10 sessions | +0pp | Refuted |
| Surprise-driven gap detection | >=70% precision | 13.8% | Refuted |
| Prediction calibration | ECE <15% | ECE = 45% | Refuted |
| Trackable infrastructure | 136 tests pass | 136 pass | Validated |
The plumbing worked. The cognition was fundamentally broken.
In a traditional process, this is a disaster. Months of work, core idea doesn't hold up, quietly shelve it. But we had something better than working code — we had specific, documented failure modes. Each failed hypothesis pointed to a concrete structural problem. No learning over time because there were no competing prediction schemas. Low precision because there was no mechanism to detect cognitive distortions. Poor calibration because there was no distinction between fast intuitive responses and slower reflective ones.
Then the insight that made the failures worth more than any success: each failure mapped to a known mechanism from Beck's Cognitive Behavioral Therapy framework. Clinical psychology had already solved these problems — just not for AI systems. The failures didn't tell us what was wrong. They told us where to look for answers.
We redesigned the architecture, formulated eight new hypotheses grounded in CBT, and published the paper as an honest failure-and-redesign story. The failures were the contribution.
Without HDD: build for months on conviction, discover problems too late, nothing to show for it. With HDD: five tests in days, exact failure modes identified, immediate redirect.
Git as the Arbiter of Truth
Pre-registration is the heart of HDD and the part that makes people uncomfortable.
Before running an experiment, you write a file specifying: the hypothesis, the outcome measure, the target threshold, the analysis plan. Commit to git. Record the hash. Then start collecting data.
If someone challenges the results — "did you really set that target before you saw the data?" — you point to the commit hash. Hypothesis committed at 2:00 PM. Data collection started at 2:15 PM. Timestamps are immutable. Git is the arbiter.
You said 15% and got 12%? That's a refutation. Not "well, 12% is close." Not "the target was aggressive." A refutation. Document what you learned, create a new hypothesis with a refined target, test again. That discipline is what separates HDD from post-hoc analysis.
The Critical Feedback Loop
Here's what ties HDD back to shipping software. Every experiment runs on a feature branch containing the implementation being tested:
HYPOTHESIS -> EXPERIMENT (on branch) -> RESULTS
|
VALIDATED -> Merge -> improved software
REFUTED -> learn -> NEW IDEA -> iterate
Only validated enhancements ship. We don't merge code we can't prove works. This keeps the codebase clean and ensures every feature has evidence behind it. Papers document what worked and why. They are proof, not the goal.
What We've Learned After 18 Months
Over 29 internal research papers and 80+ formal measures across our NuSy platform:
Pre-registration prevents wishful thinking. Before HDD, we'd run the system, look at the data, and declare success based on whatever looked good. Now we state targets first. It's uncomfortable. It's honest.
Negative results are your most valuable data. The 4/5 failure on predictive processing was more productive than many successes. If we'd declared victory based on the one passing hypothesis, we'd have shipped broken cognitive architecture and spent months wondering why.
Speed demands structure. Agentic development is fast — in our fleet, parallel Claude Code agents routinely ship multiple features in a single morning. That speed is intoxicating and dangerous. HDD prevents shipping conclusions before the evidence is in. Without it, you drown in unverifiable claims.
Metrics should be boring. Dashboards should show pass rates and target deltas, not narratives. If you need excitement to validate your AI system, you don't understand your system well enough.
Who Is This For
Any team building AI systems where "does it work?" is harder to answer than "does it compile?" If you're fine-tuning models and eyeballing outputs. If you're building RAG pipelines and guessing at retrieval quality. If you're running agents and hoping they do the right thing. HDD gives you a framework to know — not hope.
It's also for teams that need to show their work. If you're reporting to investors, regulators, or enterprise customers, "we tested it and it seemed fine" doesn't cut it. A git-backed experiment log with pre-registered targets and timestamped results is evidence. It turns "trust us" into "here's the commit hash."
The full methodology reference covers the lifecycle, experiment design, measure categories, and tooling integration. yurtle-kanban is the open-source CLI that makes the workflow frictionless — hypothesis creation, experiment tracking, pre-registration, and dual-board management in a single tool.
Adopt as much or as little as fits your workflow. The minimum viable HDD is three steps: state the target before collecting data, accept refutations as learning, and keep the cycle short.
A hypothesis that fails tells you something. A hypothesis you never test tells you nothing.