How I built AI memory (and accidentally built two)
By Kaia Colban
Claude is great, but for it to actually help me it has to understand who I am and what I'm working on in detail, not just the website copy. And I was sick of re-explaining who I am, what I'm building, and how everything works, every single session.
The advice I got depended on who I asked:
- Non-technical people said to just toggle on "Generate memory from chat history," but I haven't found that to be sufficient.
- Technical people told me to save everything to a git repo and have Claude read it. I'm not that technical, so I got lost, and when I dug in, the git-repo route clearly fit Claude Code, the coding tool, better than Cowork, where I actually do most of my work. And just saving every thread to git wouldn't solve my problem anyway, since that creates way too much context for Claude to actively read through.
So I spent an afternoon building my own memory with Claude. Here's what I built, where it broke, and the setup I actually run now.
What I built
It's one folder of markdown files, layered so a session only loads what it needs instead of everything at once:
- Identity, always on: CLAUDE.md. Short and stable. Who I am and how I want the AI to work (be concise, show sources, my writing style).
- Index, always on: MEMORY.md. A one-line-per-topic table of contents that points at the detail files. I keep it small on purpose.
- Detail files, on demand: the product brief, decisions log, lessons, projects. These only load when the topic actually comes up.
- Living logs, searched not loaded: dated decision and lesson logs I search before re-deriving something.
- Raw capture, background: the full session transcripts, saved automatically, as a last-resort backstop.
The same thing as a picture, sorted by when each layer loads:
what loads, and when
--------------------------------------------------
ALWAYS (every session, so kept small)
CLAUDE.md -> identity + working rules
MEMORY.md -> index, one line per topic
--------------------------------------------------
ON DEMAND (only when the topic comes up)
detail files -> brief, decisions, lessons
--------------------------------------------------
ON SEARCH (before re-deriving something)
living logs -> dated decision + lesson logs
--------------------------------------------------
BACKGROUND (the backstop, rarely read)
raw transcripts -> full sessions, auto-saved
The always-on layers are the ones I obsess over, especially the index. I keep it under 25 KB, because a big index isn't just token-heavy, it actually stops the model from fully digesting the file, so it loses information and uses it wrong.
Every day a capture job runs that reads the day's sessions and saves only what clears a strict bar, a real decision, recurring context, or a genuine lesson, and drops the rest. The setup itself was fine... or so I thought.
Where it broke
I realized my system was broken when my monthly cleanup task ran in Claude Code, the scheduled pass that keeps my index from bloating. After it ran, I asked Cowork "what was my memory size before and after the cleanup?", assuming that task had cleaned up the whole system.
I was lost and confused until I figured out Cowork and Code weren't communicating. Turns out, I'd set up two stores, Store A for Cowork (the desktop agent, which auto-loads its store in my Documents folder) and Store B for Claude Code (which loads its own config location). Each kept its own memory, and the cleanup I'd scheduled only touched Store B.
The real issue was that this also meant the two stores were saving and building separate memories. They both had a copy of a few of the same facts (my identity and how I describe the product), but everything else, the detail files, the logs, the raw transcripts, sat siloed in whichever store it got saved to. The whole point of my memory is for both tools to be up to date and know everything, and instead each one only knew half.
So I got to work... (meaning I got Claude to work on a fix). My first idea was to just merge the two stores with a symlink. That turned out to be a bad idea for a few reasons: symlinks don't actually give you one shared git history, my product, GTM, ops, and founder notes don't belong in the same file as my engineering notes anyway, and it added more moving parts than the problem I started with. So I scrapped it and did something simpler.
What I run now
I stopped trying to force them into one and kept the two stores separate on purpose. It's not a setup I'd pick from scratch, it comes from one hard constraint, and it's worth being precise about what the constraint actually is:
The constraint isn't which folders each tool can read. It's where each tool auto-loads a file from with no action from me. Cowork auto-loads its own store in Documents (Documents/Claude-Memory). Claude Code auto-loads its CLAUDE.md chain in ~/.claude and can pull in any file by absolute path with an @import line. There's no single folder both tools auto-load on their own, so there are two stores.
(One thing I got wrong at first: Cowork isn't locked to Documents. I can grant it any folder I want. But granting read access doesn't make Cowork auto-inject that folder into every session, and auto-load is the whole point of a memory system. So the folder my shared file lives in has to be one Cowork already auto-loads, which is its native Documents store.)
They also cover two different altitudes: Store A is founder, GTM, and customer stuff; Store B is code and engineering. Keeping those apart is a feature, not a bug. I don't want billing-PR details cluttering up my customer notes, and cramming everything into one place gives Claude too much to hold at once, which is exactly when it starts losing track and getting things wrong.
So both stores are backed up and cleaned on a schedule, and I added one small file that both of them read. Here's how it works now:
Cowork Claude Code
(auto-loads its Documents store) (loads its own config + @imports)
| |
v v
STORE A: founder brain STORE B: engineering brain
Documents/Claude-Memory/ ~/.claude/.../memory/
positioning, customers, PRs, dev setup,
decisions, lessons tooling gotchas
\ /
\ shared-context.md /
+-----> (one file) <------+
Cowork auto-loads it natively (it lives in Documents);
Claude Code pulls it in with one @import line.
Store A: the Cowork store (my "founder brain")
- Where: /Users/kaia/Documents/Claude-Memory/
- Who reads it: Cowork auto-loads this at session start. Claude Code can read it too (it's just a normal folder).
- What's in it: the stuff I keep re-explaining: product positioning, customer and discovery notes, decisions I've already made, lessons, in-flight projects, reusable prompts. This is my business-side memory.
- Files: MEMORY.md (the lean index and entry point), lore-context.md (the master product brief), decisions-log.md, lessons.md, active-projects.md, prompt-library.md, plus an archive/ folder for working files that aren't memory.
- Maintained by: a daily "curate" task, a twice-monthly cleanup pass, and me.
- Backup: its own private git repo with a private remote on GitHub.
Store B: the built-in Claude Code store (my "engineering brain")
- Where: ~/.claude/projects/.../memory/ (a symlink into my dotfiles repo), plus the CLAUDE.md identity file and domain packs and living logs under ~/.claude/domains/.
- Who reads it: Claude Code loads this at session start. Cowork doesn't auto-load it (it's outside Cowork's native store).
- What's in it: my engineering-side memory (how I ship PRs, code-review habits, local dev setup, tooling gotchas) plus identity and workflow rules.
- Maintained by: a monthly cleanup task, the /save and /recall commands, and me.
- Backup: tracked in my dotfiles git repo (private remote). One caveat I found while fact-checking this: ~/.claude isn't a git repo at its root, only some subdirs symlink into dotfiles, so the engineering side is actually my weaker-backed store. Getting ~/.claude/domains/* fully into git is on my list.
The bridge: one shared file both tools read
I built one small file, shared-context.md, that both tools read. It lives inside Store A (Documents), so Cowork auto-loads it natively, and Claude Code pulls that same file in with a single @import line in its home router. One file, loaded automatically by both.
What makes it safe is which way it points, and that follows directly from the auto-load constraint. The tempting move is to put the shared file on the engineering side (~/.claude) and just grant Cowork access to it. That doesn't work: grant-access gets you on-request reading, not auto-load, so I'd be back to symlinking out of Cowork's sandbox, the fragile direction I already scrapped (and have a scar from). Instead the shared file lives in Documents, where Cowork loads it natively, and Claude Code reaches in with one @import line. @import takes any absolute path and doesn't lean on a sandbox grant, so it's the sturdy direction. Nothing moves, and it's easy to undo: delete the file and that one import line and I'm back to before.
Only the stuff both tools actually need goes in it: who I am, my writing rules, current product positioning, and a short "current focus" snapshot. So my rule is simple: if something should be known in both tools, it goes in this file. Everything deep or tool-specific stays in its own store.
Capture writes daily, cleanup prunes monthly... all of it runs automatically, without me having to do or remember anything. It's really four separate jobs working on each store, each on its own clock, not one after the other:
FOUR JOBS, EACH ON ITS OWN CLOCK
CAPTURE (daily)
saves what clears the bar
RECALL (each session)
loads identity + index + bridge
CONSOLIDATE (scheduled)
merges, de-stales, prunes the index
BACK UP (every change)
commits to a private git repo
the shared bridge runs across all of it:
anything both tools need goes in it, and both load it
Pros and cons (honest)
Pros
- No re-explaining. Context carries across sessions on its own.
- Loads automatically. The identity file, the index, and the shared file all load at session start. Nothing to attach.
- Right-sized. The layering keeps the always-on part small and fast, and detail only loads when I need it.
- Shared where it counts. The bridge means the common core is actually in both tools, not stuck in one.
- Versioned and recoverable. Git gives me history, an undo, and an offsite copy of every change.
- Self-maintaining. The scheduled cleanup keeps it from bloating and going stale.
Cons
- Two agents can collide. If two of them edit the same store at once (the bridge is the most exposed, since both tools write to it), one can quietly overwrite the other. Nothing actually enforces this, my rule is just one writer at a time, and the git snapshots are what save me when I slip.
- Drift is the real risk. Two unsynced stores means the same fact can get copied into both and then edited in only one. My rules for this: one fact, one place (cross-tool facts live only in shared-context.md, never also pasted into a domain pack), keep the bridge short so there's less to drift, and have the cleanup pass diff the two stores so any duplicated fact gets flagged instead of silently going stale.
- The keep-bar is a judgment call. It's an AI deciding what matters, so it sometimes saves junk or drops something I'd have kept, and I still skim it now and then. And because the stores aren't unified, it's not just judging what to save, it's judging what's worth saving to both tools and pushing that into the bridge. When it misses, I have to remember to do that myself.
- It's a lot of moving parts. Numerous scheduled tasks, two git repos, a symlink, and an import line. I need to make sure nothing breaks.
If you want to build your own
You can have something working in an afternoon, and the easiest way is to let Claude build it for you. Give Claude this whole post, then paste the prompt below:
I want the memory system described in the post above. Build it for me, adapted to my setup, and explain each step in plain language as you go.
The short version: it's a small folder of markdown files you read at the start of every session, a short identity file, a lean index (one line per topic, kept under ~25 KB), and a few detail files (decisions, lessons, projects) you only open when they're relevant. Keep it lean, not a dump of everything. Please:
- Ask me which AI tool I'm using and where it can reliably auto-load files, then create the memory folder there.
- Interview me to fill in the identity file: who I am, what I'm working on, and how I want you to behave.
- Set up the index and the detail files.
- Make sure you'll actually load this in future sessions. Tell me exactly what to do so a fresh session reads the identity file and index on its own (for example, what to add to my saved instructions or settings), because a new session won't do it automatically.
- Put the folder under git so I have history and an undo, and walk me through it if I've never used git. Use a private remote if it holds anything sensitive.
- From now on, when I say "remember this," save it to the right file, but only if it's a real decision, recurring context, or a genuine lesson, never raw transcripts.
- Set up a cleanup that merges duplicates, fixes stale facts, turns relative dates into real ones, and prunes the index back under ~25 KB. If my tool can run scheduled tasks, schedule it; if not, tell me when to run it myself.
- When you're done, show me the folder, confirm you can read it, and give me a short summary of how to use it day to day.
If I use two AI tools, also set up the bridge from the post: one small shared file that lives where my more-restricted tool auto-loads it, and the other tool imports by absolute path, holding just the facts both tools need.
One heads-up: this prompt gets you the core system, but with manual capture, meaning you tell Claude when to save something. The fully hands-off version I actually run, a daily job that reviews my sessions and saves what matters on its own, plus a cleanup that runs itself, takes more setup and depends on your tool. Those are separate prompts. DM me and I'll send them over.