This is the final post in a three-part series on our approach to AI in GraphAware Hume. It explores how we’re thinking about control, transparency, and the continued evolution of Document Intelligence — an upcoming feature currently in active development.
Part 1 explored why AI in intelligence analysis should support the analyst rather than replace them.
Part 2 examined why a typical GraphRAG approach to document analysis proved unsuitable for high-stakes environments, and what we’re doing instead.
I ended the previous post with a number I hadn’t yet explained: 80.5% accuracy on a rigorous benchmark of complex, professional-grade analytical questions. The agentic retrieval architecture I described got us to 74%. The remaining gain came from something different, and I think it’s the most interesting and consequential part of what we’re building.
It came from memory.
Not memory in the way that word is sometimes used loosely in AI marketing. Nor a system that remembers what you asked five minutes ago. I mean something more substantive: a system that accumulates operational experience over time, distils that experience into explicit guidance, and applies that guidance to new questions, getting progressively better the more it’s used.
The problem with a fresh start every time
Consider the following scenario:
An analyst asks a question. The system retrieves relevant passages, evaluates them, tries different approaches if needed, and constructs a verified, evidence-linked answer. All of that work happens in the context of that single question, and then, from the system’s perspective, it’s gone.
When the next analyst asks a similar question, the system starts again from scratch. The same retrieval strategies get tried. The same early dead ends get hit. The same refinements get discovered.
All the operational knowledge earned — the understanding of what works for this type of question, against this type of document, in this organisation’s investigative context — evaporates between sessions.
This would be strange behaviour in any professional environment. An experienced analyst who has answered a hundred questions about financial networks doesn’t approach the hundred-and-first as if they’ve never done it before. They bring accumulated understanding: which indicators to look for first, which sources are most reliable for which entity types, and what constitutes a meaningful pattern versus a coincidence in this specific domain.
The question is: can a system accumulate something analogous? And can it do so in a way that’s transparent, auditable, and controllable?
Two things that need to be stored: the context graph and the evolving playbook
To understand how we’re approaching the concept of memory, it helps to distinguish between two kinds of information worth retaining.
The first is a detailed record of what happened: which question was asked, which retrieval strategies were tried, what each attempt returned, how the system evaluated the results, what the final answer was, and what the analyst thought of it. This is the raw trace, granular, complete, and valuable as an audit record. We call this the context graph.
The second is distilled guidance. Not the raw trace, but what you would write if you looked at a hundred traces and tried to summarise what they teach you. When investigating this type of question, start with this kind of search. When the analyst flags an answer as incomplete, the missing information is typically in this source category.
We call this the evolving playbook, where the context graph grows continuously and stores everything. The playbook is curated, periodically distilled from the context graph, refined, and consolidated. It doesn’t grow infinitely. It gets better.
How the evolving playbook works
When the system answers a question, the execution trace will be recorded in the context graph. If the analyst provides feedback, it will also be recorded.
Periodically, a separate process will review recent traces, analyse patterns across successful and unsuccessful sessions, and incrementally update the playbook. The updates will be specific: a new guidance entry for a question type the system has consistently struggled with, a refinement where accumulated experience has clarified what works, and a deletion of guidance that turned out to be unhelpful.
Critically, these updates will be expressed in plain language. Not as statistical weights in a model, not as numerical parameters buried in infrastructure, but as readable, specific operational guidance of the kind a senior analyst might write in a training document.
When a new question arrives, the system will identify the question type, retrieve the relevant playbook guidance, and use that guidance to inform its approach from the first iteration onward. The analyst still reviews every answer. The evidence linkage is still there, as is the audit trail. But the system starts from accumulated operational wisdom rather than from zero.
This accumulated, applied, domain-specific experience is where our additional gain in accuracy comes from.
Why transparency matters more than accuracy
The improvement in accuracy matters, but it’s actually a less important part of the story. The more important part is what this kind of learning mechanism makes possible for organisations with strict governance requirements.
The standard concern with AI systems that “learn” is that they learn in ways that can’t be seen, explained, or controlled. An auditor or a court asks, “Why did the system give this answer?” and the honest answer is, “We’re not entirely sure.” That’s not a defensible position.
Our design takes the opposite approach. The playbook will be a document your organisation can read in full at any time. Every entry will have a lineage, traceable back to the context graph entries that generated it. An analyst can read the playbook and disagree with an entry. A supervisor can review and correct it. An administrator can delete entries, add entries, or freeze the playbook entirely if the organisation prefers a static configuration. Furthermore, when a playbook entry is used for a question, it is also traced to evaluate the relevance of the entries and ensure traceability.
Nothing in the playbook will be hidden. Nothing will be derived from a process that can’t be explained. It will accumulate operational wisdom in a form that any qualified person can inspect, challenge, and override.
What the system will not learn
The system will not learn from the content of your documents in a way that persists beyond your organisation’s environment. It won’t develop opinions about suspects, incorporate assumptions from previous cases into new ones, or build up a model of any individual based on accumulated exposure to case material.
The context graph records the process: how questions were answered, what retrieval strategies worked, and so on. It doesn’t form substantive conclusions about the people or organisations your investigations concern.
The playbook captures the operational methodology: how to search effectively for specific types of information, how to interpret certain kinds of results, and what quality criteria an answer must meet. It doesn’t capture or propagate investigative conclusions. An analyst’s judgement remains entirely their own. The system simply helps them exercise that judgement more effectively.
This distinction between a system that learns to work better and one that learns to reach its own conclusions is one I feel strongly about. It’s the difference between a tool that augments professional expertise and one that displaces it. The architecture we’re building enforces that distinction structurally, not just as a policy statement.
Where this fits our broader vision for AI-enhanced intelligence analysis
I’ve described three sets of capability across this series: Document Intelligence, iterative evidence-linked retrieval, and the evolving playbook.
They aren’t three separate features. They’re one coherent approach to a single question: what does responsible, genuinely useful AI look like in a domain where the consequences of getting things wrong are high, where auditability isn’t optional, and where the analyst’s judgement is irreplaceable?
The answer we’ve arrived at is an architecture built around four commitments:
- Every AI output must be grounded in the organisation’s own verified data, not in the model’s general knowledge or training patterns.
- Every output must be linked to the specific source that supports it.
- The analyst must initiate, approve, and own every output; the AI should not make decisions autonomously.
- The reasoning behind any result must be visible and auditable in a form that a working analyst or an external auditor can actually inspect.
These are the non-negotiable architectural constraints we have to work within to be a reliable, responsible partner to our customers.
No system is foolproof. There will be cases where the system misses evidence or where the playbook develops a heuristic that proves wrong for a specific document type. But that’s why we place such importance on evidence linkage. It’s the key to preventing errors because it makes errors visible, traceable, and correctable rather than silent and persistent.
I’m confident in our direction: AI that works with the analyst’s judgement rather than replacing it, that earns trust by showing its work, and that gets better over time in ways the organisation can see and control.
That’s what we’re building. This series has been an attempt to explain, as honestly as I can, how and why.
GraphAware Hume is a graph-powered intelligence analysis platform used by law enforcement and intelligence agencies. Document Intelligence is currently in active development. If you would like to follow the progress of this work or discuss how it applies to your organisation’s needs, get in touch.
