Building shared memory for my AI tools

I use Claude Code, Claude, and ChatGPT for the same projects. Sometimes I’m working on my Mac, sometimes I’m talking through an idea on my phone. And every time I switch, I have to explain things I’ve already explained somewhere else. What the project does, what we tried, why we decided against something. The code is in the repo, but a lot of the thinking is in those chats.

In July, I started building Personal Ledger with Claude Code. Basically, I wanted to put my conversations in one place and let any of these tools search them through MCP. I didn’t want to pick one assistant and stay there just because it had more of my history. Keeping a separate summary up to date for every project didn’t sound appealing either.

Getting the history into a database was only part of it. Once I started asking about my projects, the ledger sometimes returned an old decision with more confidence than the one that replaced it. It had the conversation where I changed my mind. It was still giving me the old answer.

I wanted to use it from my phone too

One of my early questions was where the database would live. Running everything locally was fine while building it, but I also wanted to use it from my phone with my Mac turned off. So the storage went on AWS, with an MCP server that the assistants could connect to.

The inputs needed different ways in. Coding sessions could send turns through a hook and a local outbox. Old chats came from account exports. For new conversations in Claude, I asked for a tool that could save a note directly through MCP. That let me save something without waiting for browser capture, although a note is only as complete as whatever the assistant decides to put in it.

I also wanted the files. If I’d uploaded a PDF and spent a conversation discussing it, saving “look at this PDF” without the PDF wasn’t going to help much later. Documents go into storage under a hash of their contents, and extracted text keeps references to the page or part it came from. If an attachment is missing, the importer records that instead of treating the conversation as complete.

Keeping the conversations

I wanted to keep the original conversations even if the summaries were wrong. Each imported message goes into an event ledger, where it can’t be edited or deleted. From there, normalizers build conversations and message revisions, and an extractor reads those conversations to produce shorter memories. If the extraction turns out to be bad, I can go back to the messages and run it again.

01 / How the pieces connect

Scroll to inspect · tap to open full size

The database enforces this by rejecting updates and deletes on the event ledger. Correcting a memory changes the derived record; it doesn’t rewrite what was said in the conversation.

The same message can come in more than once. A turn might arrive through a live hook, then again in an export or a retry. A SHA-256 content hash and a unique constraint catch duplicates, as long as the input is normalized consistently. The hash includes timestamps and metadata too. Give the same message a fresh timestamp on every import and it will look like a new event.

Finding a chat when I don’t remember the wording

At one point I tried the connector and found out semantic search wasn’t there yet. We had the ingestion part, but I was already asking questions without remembering the exact words I’d used in the original chat. Being able to store everything wasn’t much use if I still had to know what to search for.

Exact lookup is still useful for an ID or a filename, and PostgreSQL full-text search handles words I do remember. For the rest, there’s pgvector. The hybrid search runs text and vector retrieval, then combines their results with reciprocal rank fusion:

$r_\ell(d)$ is the result’s rank in list $L_\ell$, starting at one. Each list that contains it adds a contribution to its score. Here’s the code, with result-object construction left out:

scores = {}
for results in result_lists:
    for rank, hit in enumerate(results, start=1):
        key = (hit.kind, hit.ref_id)
        scores[key] = scores.get(key, 0.0) + 1 / (60 + rank)

ranked = sorted(scores, key=scores.get, reverse=True)

I like that this avoids having to make the text-search scores and vector similarities comparable. It uses the positions in each list. And if the query embedding is unavailable, full-text search can still return results.

But both searches can do their job and still leave me with the wrong context. The conversation where I chose an approach and the conversation where I abandoned it are both relevant to the same project.

It remembered a decision I had already changed

For an assistant starting a task, I wanted something shorter than a pile of search results. The ledger extracts statements from conversations and puts them in a context pack, with references back to the evidence. The idea is that the assistant can get caught up without reading every chat first.

When I checked what was in one of these packs, it had an old project status and a newer one. The old statement had higher confidence. Some memories were repeated too. My reaction during debugging was: “why not just retrieve the recent memories??”

That helps if the last thing saved is the actual decision. But the last thing saved can also be an assistant repeating something outdated. Here’s a made-up example of the problem:

I’d want Tuesday’s decision here. Wednesday adds no new information; the assistant is repeating Monday. A high confidence score on Monday’s memory doesn’t help either. The extractor can be confident about what I said and still be giving me a decision I’ve changed.

The fix: mark the old decision as superseded

The fix is to record that one memory replaces another. The old one gets status = 'superseded'. The replacement stays current and points to the old memory through supersedes_memory_id. The context pack leaves out superseded memories. It also checks authority, so an unsupported assistant assertion doesn’t get treated as a current fact.

After resolving that example, this is what should be left:

These are the two filters in the context-pack query. I’ve left out the search and project-scoping conditions:

SELECT memory_id, statement, authority, last_confirmed_at
FROM memory_items
WHERE status = 'current'
  AND authority != 'asserted';

Now Postgres is the only eligible memory. I can still search for the Redis discussion if I want to know how we got there. It just stops being part of the answer to what we’re using now.

Catching the change automatically

Of course, those fields have to be set correctly first. During extraction, the system finds the closest existing memory by embedding similarity. At a cosine similarity of 0.86 or higher, it treats the new statement as a duplicate. If the closest match is current and the score is at least 0.70 but below 0.86, it asks an LLM whether the pair is duplicate, supersedes, or distinct. A supersedes result creates the link and changes the old status. Repair passes do similar checks on memories already in the database.

I’m not fully happy with this part. A changed decision can score above 0.86 and get discarded as a duplicate without reaching the LLM. A related statement below 0.70 misses the comparison entirely. And even inside that range, the LLM can get it wrong. If it misses a replacement, the old memory stays current and passes the SQL filter we just added.

So I also need a way to correct it myself. kb_correct_memory takes the old memory ID and my corrected statement. It saves the correction as a source event, creates a current, user_confirmed replacement, and marks the old memory superseded. There’s no LLM deciding whether to accept that correction:

02 / What an explicit correction changes

Scroll to inspect · tap to open full size

I asked whether anything had been deleted while we were fixing this. Both versions and their evidence are still there, which is what I wanted. I might disagree with the old answer and still want to read the discussion that produced it.

The source was the assistant’s own note

Another bad memory led back to a QA note the assistant had written. The extractor had read the note and turned it into a fact. I followed the source ID expecting to find the conversation that supported the claim, and got the assistant’s own summary.

The assistant saves a note, the extractor turns it into a memory, and the next answer cites that memory. Save another note from that answer and the same claim comes around again:

03 / When a summary becomes evidence

Scroll to inspect · tap to open full size

This is why the query excludes asserted authority. The extractor now skips streams made entirely of manually captured notes, and a repair pass retracts existing memories supported only by those notes. The notes themselves remain searchable. Some of them are useful summaries; I just don’t want their existence to count as evidence that the assistant was right.

Debugging added another awkward source of bad memories. I’d show the assistant an incorrect statement and ask it to fix it. Now the bad statement was in a conversation too. If extraction picked up the quote and missed the fact that I was disputing it, we could end up saving the same error again.

For now, context packs exclude memories from the ledger’s own development sessions unless the query is about the ledger. That exception uses keywords, so rephrasing a query can change what gets through. It also doesn’t solve the general problem of telling a quoted claim from something I actually believe.

The current answer disappeared too

After filtering out weak evidence, I checked again. The misleading result was gone, but so was the pack’s only account of the project’s current state. The original conversations still had the information. Extraction hadn’t recovered a replacement from them.

So now it wasn’t giving me the old answer, but it couldn’t give me the current one either. A test that only checked whether the bad memory disappeared would have passed. I needed to check that the replacement was there too, and that its evidence led back to the original conversations.

Where I am with it

I used the ledger while writing this post to find the early development conversations, including my questions about phone access and where the database would live. I didn’t remember which session they were in. That is already useful to me.

What I haven’t measured is how much it improves the assistant’s work after it gets that context. I want to start with projects where I changed a decision and check whether the pack returns the replacement with the right evidence. I also need cases where two similar statements are both valid, because automatically marking one as outdated would make things worse.

For now I still open the source when a remembered fact matters. I can correct a bad memory without losing the conversation, but I don’t yet trust the extraction to catch every change on its own.

← All writing · Code on GitHub · Substack ↗