Thinking about agent memory as 4D space (Part 1 | Theoretical)
Without memory, a model responds only to the information immediately in front of it. A stateless invocation, without nuance . An effective memory strategy, can connect the current request to previous decisions, preferences, outcomes, and learned ways of working.
But more memory does not automatically create a better agent. Poorly designed memory introduces noise, stale assumptions, conflicting facts, and retrieval overhead.
The question you should be asking is not:
How much memory should the agent have?
It should be:
What should the agent remember, for how long, in what form, and with which relationships preserved?
Let’s began thinking about memory across four dimensions:
- Time: How far back should the agent remember and weighted time of the contet?
- Space: Across how many locations is memory distributed, isolated locations or a single raw context?
- Summarisation: How much has the information been compressed, how to determine importance?
- Structure: How do memories depend on and connect to one another, is it Independent, Linearly dependant or a Graph?
The Fourth Dimension
A memory strategy is not a single technology choice. An agent may simultaneously use:
- Recent events;
- Summaries from the last month;
- Procedural rules learned across tasks;
- Explicit enterprise rules
- External organisational systems;
- Relationships between items or events.
Each source requires a different stratergy of time, space, summarisation, and structure.
The goal is not to maximise every dimension. It is to place each type of information where it creates the most value at the lowest cost. Where context is stored as efficient markers of valueable data, to help improve and constrain models within bounds or scope.
Dimension 1: Time
How far back should the agent remember?
Time describes the historical depth available to the agent.
Older memories may be compressed, given a lower retrieval weight, replaced by newer information, promoted into persistent knowledge, or deleted.
Consider a coding assistant conversation:
Sarah: I'm getting a CORS error on the payments API.
Agent: The API is missing the Access-Control-Allow-Origin header.
Sarah: That fixed it. I also dislike verbose error messages.
Sarah: My team uses tabs rather than spaces.
During the task, I want the full conversation to remain in transient memory.
After a period of inactivity, it can be compressed:
{
"summary": "Resolved a CORS issue on the payments API.",
"facts": [
{
"key": "communication.error_messages",
"value": "concise",
"confidence": 0.9
},
{
"key": "coding_style.indentation",
"value": "tabs",
"confidence": 0.9
}
]
}
After 30 days, the conversation summary may be removed while the useful facts remain. If Sarah repeatedly confirms the use of tabs, the preference may become a persistent rule:
{
"trigger": "generate_code",
"condition": "user_id == sarah",
"action": "use_tabs_for_indentation"
}
The CORS episode may eventually expire because it was a one-off. The stable preference survives because it continues to affect future behaviour.
Time is also a trust problem. An old preference may still be valid, while an old system status or project assignment may be dangerously stale.
Useful metadata includes:
- first observed;
- last confirmed;
- confidence;
- expected decay;
- whether a newer memory supersedes it.
The design question is:
Would reaching further into the past provide useful context or outdated assumptions?
Dimension 2: Space
Across how many locations is memory distributed?
Space describes how many memory locations the agent must query.
{
"preferred_language": "English",
"response_style": "concise",
"programming_language": "Python"
}
For the implementation, I used a partitioned approach:
Current conversation → Working context
Recent sessions → Summary memory
Stable facts → Semantic memory
User choices → Preference memory
Past tasks → Episodic memory
Rules and relationships → Self-managed memory
A coding agent may also use:
- the current file in context
- a vector store for code chunks
- a symbol index for definitions and references
- project preferences
- configuration, libraries and linting rules
An enterprise agent may retrieve from documentation, tickets, customer records, ownership data, incident systems, and live APIs.
The more distributed the memory, the more important orchestration becomes:
- Which source should be queried?
- Which source is authoritative?
- What happens when sources conflict?
- Which systems may the agent access?
- How much latency is acceptable?
The design question is:
Is the agent retrieving from one brain or coordinating a council of specialists?
Dimension 3: Summarisation
How much of the original experience should survive?
Summarisation describes how much information has been compressed or refined.
Each transition removes detail but increases reuse.
For example:
Raw: "My team uses tabs rather than spaces."
Extracted: "coding_style.indentation = tabs"
Distilled: "When generating code for Sarah, use tabs."
Crystallised: "The formatter automatically applies tabs."
In the code, I use summary memory for L1, semantic and preference memory for L2, episodic memory for reusable experiences, and the self-managed pipeline for promotion into L3.
Frequently used and stable memories may benefit from greater summarisation. However, promotion should require validation.
The baseline I used was:
Confidence >= 0.95
Confirmed in at least 3 sessions
Observed across at least 30 days
An incorrect transcript can be corrected. An incorrect persistent rule may silently affect every future action.
The design question is:
Should the agent remember what was said, what it meant, or how it should change future behaviour?
Dimension 4: Structure
How do memories relate to one another?
Structure describes the dependency topology between memories.
Independent
Independent memories stand alone:
{
"response_style": "concise",
"project_language": "Python"
}
One fact does not depend on the other. Semantic or preference memory is normally sufficient.
Dependent
Dependent memories rely on order, cause, or conditions:
Deployment failed -> image could not be pulled ->
repository permission was missing -> update the runtime role ->
retry deployment
Procedural memory is often dependent:
When adding an API endpoint -> define schemas -> add authorisation ->
add tests -> regenerate the client
In the implementation, episodic memory captures the task, approach, outcome, and lesson rather than storing only the topic.
Graph
Graph memory represents typed relationships:
Sarah -> works_on -> Payments
Payments -> owned_by -> Backend Team
Backend Team -> managed_by -> Bob
This allows the agent to infer that Sarah’s payments issue should be routed to Bob’s team.
The self-managed pipeline can extract relationships such as:
works_on
owns
depends_on
reports_to
uses
affected_by
supersedes
For smaller systems, these can remain as memory records. For traversal-heavy use cases, they can be projected into a graph database. Beware can be costly.
The design question is:
Does the agent need to know that something is true, how it happened, or how it connects to other things?
Mapping the Framework to the Code
The implementation uses five memory strategies:
StrategyPurposeSummaryRecent conversational continuitySemanticStable facts and decisionsUser preferenceDurable user choicesEpisodicPrevious tasks, outcomes and lessonsSelf-managedPromotion, decay, conflicts and graph extraction
These strategies are complementary.
Summary memory answers: What happened recently?
Semantic memory answers: What became true?
Preference memory answers: What does the user consistently want?
Episodic memory answers: What happened before, and what did we learn?
Self-managed memory answers: What should be promoted, connected, deleted, or treated as stale?
A Tiered Memory Lifecycle
The framework can be implemented as rolling memory windows.
10 messages
OR 8,000 tokens
OR 120 minutes of inactivity
A possible transition policy is:
L0 → L1
Summarise the task, decisions and outcome.
L1 → L2
Merge repeated facts, preserve episodes and flag conflicts.
L2 → L3
Promote stable, repeated and high-confidence information.
Information that is old, low-confidence, and never reinforced should be deleted.
The code uses the following baseline:
Delete when:
Older than 365 days
AND confidence < 0.60
AND fewer than 2 confirmations
Conflicting memories are preserved with their source, date, and confidence rather than silently overwriting one another.
Diagnosing Memory Problems
Memory is a Managed Transformation
What I wanted to build was not an ever-growing transcript.
I wanted a system that continuously transforms information:
Experience -> Summary -> Fact or episode -> Relationship or procedure ->
Behaviour
Some experiences should remain temporary. Some should survive as searchable episodes. Some should become persistent facts. A small number should eventually influence behaviour by default.
The best memory architecture is not the one that remembers the most. It is the one that deliberately decides what should be preserved, compressed, connected, promoted, or forgotten.