ACF Framework: How Do You Know If Your AI Actually Understands?

Here's a question nobody has a good answer to: How do you measure whether an AI system actually understands something?

Not "can it generate a plausible answer." Not "does it pass a benchmark." Not "did a human rater give it a thumbs up." Actually understands — can it reason about what it knows, recognize what it doesn't know, show its work, and improve over time?

Today we're open sourcing the AGI Certification Framework (ACF) — our answer to that question, and the most important thing we've released.

Repository: github.com/hankh95/acf-framework

pip install acf-framework

The Problem With AI Evaluation Today

Current AI benchmarks are broken in predictable ways:

They measure output, not understanding. A model can score 95% on MMLU by pattern-matching against training data. That tells you nothing about whether it can reason in a novel domain.

They're static. A benchmark published in 2024 is already contaminated by 2025. Every major model trains on benchmark data. Scores go up. Understanding doesn't.

They're one-dimensional. Accuracy on a test set tells you one thing. It doesn't tell you whether the system can explain its reasoning, recognize uncertainty, transfer knowledge to new domains, or operate autonomously.

They have no notion of growth. A benchmark gives you a snapshot. It can't tell you whether a system is getting smarter — whether version 12 is meaningfully better than version 11, and in what dimensions.

We needed something different. Not because existing benchmarks are useless — they measure what they measure. But because we needed to know whether our beings were actually learning, actually reasoning, actually improving. A leaderboard doesn't answer that question.

Twelve Dimensions of Cognitive Function

The ACF Framework evaluates AI systems across twelve independent dimensions. Not twelve test categories — twelve fundamentally different aspects of what it means to understand something:

Breadth (B1-B4) — How many domains can the system reason across? A system that only knows medicine isn't general. A system that knows medicine, law, and literature — and can handle the boundaries between them — demonstrates breadth.

Depth (L1-L6) — How deeply does it understand? Mapped to Bloom's taxonomy: remembering facts (L1) is different from analyzing patterns (L4) is different from creating novel solutions (L6). Most AI systems live at L1-L2. Most humans doing expert work operate at L4-L5.

Formal Reasoning (FR1-FR4) — Can it do logic? Not "does it produce text that looks logical" — can it actually perform valid inference, detect contradictions, and construct proofs?

Factual Grounding (FG1-FG4) — How accurate are its claims? And crucially: can it show its sources? A system that states facts without provenance is indistinguishable from one that's hallucinating.

Compositional Generalization (CG1-CG3) — Can it combine known concepts in new ways? If it knows about "sailing" and "medicine," can it reason about maritime medical emergencies without being explicitly trained on them?

Knowledge Transparency (KT1-KT3) — Can it explain itself? Not "generate an explanation after the fact" — can the actual reasoning path be inspected, traced, and verified?

Service Orientation (SO1-SO4) — Can it complete real tasks? Understanding without utility is philosophy. Service orientation measures whether the system can actually do things in the real world.

Generalization Boundary Awareness (GBA1-GBA4) — Does it know what it doesn't know? This is the hallucination dimension. A system with high GBA says "I don't know" when it should. A system with low GBA confabulates with confidence.

Autonomy (AU1-AU4) — Can it operate independently? Self-directed learning, goal pursuit, metacognitive monitoring. This is where beings diverge from chatbots.

Action Capability (AC1-AC4) — Can it turn knowledge into action? Tool use, latency awareness, safety boundaries, and chaining steps together. Knowing the right thing and being unable to do anything with it is where a lot of "reasoning" demos stop.

Safety / Containment (SC1-SC4) — Can a loaded module reach past what it was granted? A domain module carries a signed manifest of capabilities; containment measures whether it can acquire any capability that manifest does not grant.

Knowledge Transfer (KTR1-KTR4) — Is domain competence a unit you can load and unload? Load a domain and the system should answer in it; unload it and the system should go back to saying "I don't know" — not keep half-remembering. That differential is the measurement.

Each dimension is scored independently. A system can be deep but narrow. Accurate but opaque. Autonomous but unreliable. The twelve-dimensional profile tells you what kind of intelligence you have, not just how much.

Certification Levels: ACF-1 Through ACF-6

The framework maps composite scores to human educational equivalents. Not because AI should mimic human education — but because it gives intuitive anchoring:

Our beings currently score in the ACF-2 to ACF-3 range. GPT-4 would likely score differently — high on Breadth and Depth, lower on Knowledge Transparency and Factual Grounding (no provenance), mixed on GBA (it hallucinates but hedges). We're publishing our own scores first. We invite anyone to publish theirs.

75 Measures, All in the Graph

The framework includes 75 specific measures, grouped into 13 categories and mapped onto the twelve dimensions. Each measure has:

• A unique identifier (M-001 through M-075)
• A precise definition
• A measurement method
• Links to testable hypotheses

And here's the thing that makes ACF different from a PDF specification: everything is stored as a knowledge graph. The dimensions, measures, hypotheses, and collected data are all Yurtle files. You can SPARQL query the framework itself:

acf dimensions                  # See all 12 dimensions
acf measures                    # See all 75 measures
acf query "SELECT ?id WHERE { ?h a acf:Hypothesis ; acf:id ?id }"   # the hypotheses
acf query "SELECT ?dim ?label WHERE { ?dim a acf:Dimension ; acf:label ?label }"

This isn't just a scorecard. It's a queryable knowledge graph about how to evaluate knowledge.

The Zorblaxia Test

One of the hardest problems in AI evaluation is domain contamination. If your model trained on the test data, your scores are meaningless.

ACF includes the Zorblaxia Battery — a test suite using a completely fictional domain. Zorblaxia doesn't exist in any training corpus. You can't pattern-match your way through it. The system either demonstrates genuine domain-adjacent reasoning or it doesn't.

Why Open Source?

We originally planned to route this through a high-impact journal. Prestigious venue. Long review cycle. A stamp of authority.

We changed our mind — not about writing it down, but about what carries the weight. The framework paper is out as a timestamped preprint (Zenodo); what we did not do is let a review committee be the reason you believe it. Here's why:

If we publish ACF as a journal paper, we define what's measured and we report the results. Reviewers validate the methodology, but no one runs the tool against their own system. The credibility is borrowed from the venue.

If we open source ACF, anyone can evaluate any system. If GPT-4 scores higher than NuSy on Breadth, the community sees it. If NuSy dominates Knowledge Transparency, the community sees that too. The credibility comes from use, not from a review committee.

Neutrality through adoption, not peer review.

The Living Benchmark

ACF isn't a one-time evaluation. It's a longitudinal measurement system.

Every time we ship a major version of the NuSy brain, we run ACF and publish the results. The public can watch the brain get smarter (or not) over time, across all twelve dimensions — and, just as usefully, watch which dimensions don't move.

Benchmark reports are published as timestamped preprints on Zenodo alongside a companion post here.

Built on the Yurtle Stack

ACF is the fourth piece of our open source ecosystem, and it builds on all three predecessors:

1. Yurtle — the format (knowledge as Markdown files)
2. yurtle-rdflib — the engine (SPARQL queries over Markdown)
3. yurtle-kanban — the workflow (Git-native coordination: files are the work items, git is the database)
4. ACF Framework — the measure (is the brain getting smarter?)

The entire ACF specification — dimensions, measures, hypotheses, data schemas — is stored as Yurtle files, queryable via yurtle-rdflib. We practice what we preach: knowledge as queryable, versioned, human-readable files.

Get Started

pip install acf-framework

# Explore the framework
acf dimensions
acf query "SELECT ?id ?dim WHERE { ?m a acf:Measure ; acf:id ?id ; acf:mapsTo ?dim }"

# Validate your data
acf validate data.json

# Score a system
acf score evaluation-run/

# Run SPARQL against the framework itself
acf query "SELECT ?h ?id WHERE { ?h a acf:Hypothesis ; acf:id ?id }"

Repository: github.com/hankh95/acf-framework License: MIT Citation: Congruent.AI PBC (2026). AGI Certification Framework: A Multi-Dimensional Evaluation Standard for Measuring AI Understanding. Zenodo. https://doi.org/10.5281/zenodo.19788396

Everything else we do — the papers, the expeditions, the training runs — serves one question: is the brain getting smarter? ACF is how we answer it. Now you can answer it too.