Your AI memory problem is a permissions problem
Every agent memory pitch is about recall. What stopped me building one company-wide was access control, and the systems that already solve it are the ones you're already paying for.

I built an AI memory layer this year. Then I took it apart.
Not because it didn’t work. It worked fine. A database, some embeddings, a retrieval step, and my agents could pull up a decision from months ago without me pointing them at the file. For one person, it was genuinely useful.
It stopped making sense the day I sketched what it would look like for the whole company.
The question nobody puts on the slide
Going from one user to everyone isn’t a storage problem. Postgres will happily hold the vectors. The hard question is much older and much duller: who is allowed to see what?
In any real organisation, knowledge isn’t flat. The board pack isn’t for everyone. Salary bands aren’t. The draft restructure HR is quietly working on definitely isn’t. Engineering has incident notes that should stay with engineering, finance has numbers that should stay with finance, and even inside one team there are levels.
A single memory store that ingests all of that and answers questions about it has to enforce every one of those boundaries, on every query, forever. It also has to cope with permissions that change after a document was indexed. Someone moves teams. A folder gets locked down after a mistake. The source system knows within seconds. Your embedding index finds out whenever the next re-sync runs, if it runs.
That’s when I saw what I was really signing up to build. Not a memory platform. An access control system, with a memory platform bolted on the side.
The vendors are right, just not about the thing they’re selling
This market is loud right now. Mem0, Zep, Letta and a long tail of “memory layers”, plus a stack of comparison guides that are, as often as not, written by one of the vendors being compared.
A common line in those pitches is that a plain vector database isn’t enough. You need extraction, deduplication, conflict resolution, temporal ordering.
I agree you need more than a vector database. I think they’re pointing at the wrong gap.
What’s missing is the security layer, and there are really two of them.
The first is obvious. A user must never retrieve something they couldn’t open in the system it came from. That’s table stakes, and it gets harder than it sounds once you’re chunking documents and blending results from five sources with five different permission models.
The second is subtler, and I think worse. The model must not synthesise its way across the boundary. Your retrieval can be perfectly permission-filtered and you can still leak, because the “memory” the agent wrote last week was distilled from material the person asking today could never see. An extracted fact doesn’t carry the access list of the document it came from unless someone designed it to. Most memory designs I’ve looked at haven’t.
A leaked document is an incident. A leaked conclusion is an incident nobody notices.
I didn’t give up semantic recall. I moved it.
When I tore my setup down, the obvious objection was: so your agents just grep now?
Not quite. The recall moved into the systems of record.
For documents, tickets and people, it’s the work graph that sits inside the collaboration suite. In my case that’s Atlassian’s Teamwork Graph. For data, it’s the warehouse with a semantic layer on top (I use Cube), so that “revenue” or “active customer” means one thing whether a dashboard or an agent is asking.
Both of those already know who you are and what you’re allowed to see. The permission model is the one the organisation already runs on, already audits, and already fixes when it’s wrong. The agent queries as the user and inherits exactly their access. No second copy of the truth, and no second copy of the access lists drifting out of date beside it.
Is it as clever as a dedicated memory layer? No. Some questions it answers worse, mostly the fuzzy “didn’t we talk about something like this once?” kind, where nobody wrote down the word you’re now searching for. I’ll take that trade. I would much rather an agent occasionally fails to find something than occasionally finds something it shouldn’t have.
Back the vendors who’ll still be standing when the goalposts move
The other half of my thinking was vendor risk, and in this space it isn’t a box-ticking exercise.
Agentic AI is moving fast enough that the right way to store and retrieve context changes every few months. Embeddings were the answer, then hybrid search, then knowledge graphs, then long context windows made half of it look optional. Pick a young memory startup today and you’re betting it reads every one of those shifts correctly, and survives long enough to ship the response.
Google, Microsoft, Atlassian and Salesforce are fighting hard over exactly this ground. Whatever the state of the art turns out to be in retrieval and semantic layers, they’ll put it into the products you already pay for, because they can’t afford not to. And they’ll put it on top of a permissions model that has already survived a decade of enterprise audits.
That isn’t loyalty to big vendors. It’s choosing fewer things that can go wrong.
What I’d tell a CTO about to buy one
If your team wants to add a memory platform to the agent stack next quarter, start with the stack you’re already on.
Look at where your knowledge actually lives. Look at the permission model it already enforces: groups, spaces, sharing rules, row-level access in the warehouse. Any memory platform you consider has to plug into that model, not copy it. Ask the vendor what happens when someone’s access is revoked at 10am. If the answer involves the words “nightly sync”, you have your answer.
Then look at where inference is heading. Context windows keep growing and models keep getting better at working with structured retrieval. Good search over a well-organised work graph, plus a proper semantic layer for your data, may well be all the memory most agents need. A fair few of the platforms on sale today are solving a problem the next couple of model releases could quietly shrink.
Remembering is the easy part. Knowing who not to tell is the job.