Most AI tools have the memory of a goldfish. Every session starts cold. You re-explain context, re-paste documents, re-describe your business — and the model answers like it's never met you before.
We got tired of that. So we built a semantic recall system: a memory layer that stores meaning, not just text, and retrieves the right context at the right moment. It gets sharper over time because it's designed to.
What exactly is a semantic recall system?
A semantic recall system stores information as vector embeddings — numerical representations of meaning — and retrieves the most relevant chunks based on conceptual similarity, not keyword matching. Ask it about "client invoice delays" and it surfaces notes about "payment chasing" even if those exact words never appeared together. We've built this pattern into several client products now, and the core mechanic is always the same: embed on write, retrieve by similarity, rank by recency and relevance.
The difference from a basic RAG pipeline is the feedback loop. The system learns which retrievals actually helped and weights future queries accordingly. That's what makes it get smarter instead of just staying static.
Why does vector search beat keyword search for memory?
Keyword search fails the moment someone uses different words for the same idea — which is constantly. Semantic search converts both the query and the stored content into high-dimensional vectors, then finds the closest match by cosine similarity. A user asking "who handles billing disputes?" retrieves a note tagged "accounts receivable escalation" without any shared keywords.
We use OpenAI's text-embedding-3-small for embeddings. It's fast, cheap at around $0.02 per million tokens, and dimensionally stable enough that we haven't had to re-embed entire corpora when model versions update. Vectors live in pgvector, Postgres's native extension, which keeps the stack lean.
One trade-off we made early: we picked pgvector over Pinecone because we already had Postgres and didn't want to manage a separate vector DB. For collections under 10 million vectors, that's the right call. Above that, you'd want to reconsider.
How does the system actually store and chunk content?
Storage is where most teams get it wrong. Embed whole documents and you lose precision — the vector averages out across too much content. Chunk too small and you lose context.
We landed on 512-token chunks with a 64-token overlap between adjacent chunks. The overlap means a sentence split across a boundary still gets represented fully in at least one chunk. Each chunk gets stored with its embedding, a reference back to the source document, a timestamp, and a usage counter that starts at zero.
That usage counter is the seed of the feedback loop. Every time a chunk gets retrieved and the user's session produces a positive signal — they stayed in the conversation, they didn't rephrase and retry — the counter increments. Chunks with high usage scores get a small boost in future retrieval ranking. It's not ML. It's a weighted sort. But it works.
What does the retrieval pipeline look like under the hood?
A query hits the system in four steps.
First, the query text gets embedded using the same model used at write time — this is critical. Mismatched embedding models produce incoherent similarity scores. Second, we run a vector similarity search against pgvector using an IVFFLAT index, which approximates nearest-neighbour at the cost of small accuracy loss. Acceptable for this use case, and it keeps query times under 40ms at our current scale. Third, we apply a recency decay: chunks older than 90 days get a 15% score penalty unless they've been retrieved frequently. Fourth, the top-k results — usually five to eight chunks — get injected into the LLM context window as structured memory blocks.
The model sees them labelled with source and date so it can reason about relevance itself, rather than trusting the retrieval layer blindly. That labelling step was a late addition. It made answer quality noticeably better.
How does the system get smarter over time?
The feedback loop runs on three signals. Explicit feedback — a thumbs up, a "that was helpful" button — is the cleanest but the rarest. Implicit feedback — session length, follow-up questions that build on the answer rather than correct it — is noisier but abundant. Failure feedback — the user rephrasing and resubmitting within 30 seconds — is the most useful signal of all, because it's unambiguous.
We log all three against the chunk IDs retrieved for that query. Once a week, a background job recalculates usage scores across the corpus. Chunks that consistently surface for successful sessions float up. Chunks that get retrieved and then immediately abandoned get gradually deprioritised.
I've started calling it "institutional memory that earns its place" — you can read more about where that thinking comes from on the founder page.
What went wrong the first time we built this?
Honest answer: we over-engineered the feedback loop. Our first version tried to do real-time score updates after every session — recalculating weights on the fly, reordering the index constantly. It was slow. It was fragile. And it introduced a race condition where a chunk being updated mid-retrieval would sometimes return a stale score.
We scrapped it. The weekly batch job is dumber but rock solid, and the quality improvement between batch runs is imperceptible to users.
The lesson: feedback systems don't need to be real-time to be effective. Memory isn't a live dashboard. It's a slow accumulation of signal. Build it accordingly. If you're curious what that kind of deliberate trade-off thinking looks like in production, the Nuke project is a good example.
When does this architecture make sense to build?
Not every product needs semantic recall. If your users are doing one-off queries with no continuity between sessions, a stateless RAG pipeline is simpler and cheaper to maintain.
Semantic recall earns its complexity when sessions are ongoing, when users are specialists who build up domain-specific context over time, or when the cost of re-explaining context is high. We've built it into products where operators field recurring enquiries across long case timelines — the system remembers what's been discussed before and surfaces it without prompting.
We cut one client's average query-to-answer time from six minutes to under 90 seconds because the model stopped asking clarifying questions it could already answer from stored memory. That's the kind of number that makes the build worthwhile.
If you're not sure whether your use case fits, get in touch — we can usually tell within one conversation.
How do you stop the memory from going stale or wrong?
Two mechanisms. First, hard expiry: any chunk that hasn't been retrieved in 180 days and has a usage score below 5 gets archived — still stored, but excluded from the active index. Second, source versioning: when a source document gets updated, we re-embed the new version and deprecate the old chunks rather than replacing them. This means you can see the history of what the system "knew" at any point in time, which matters for audit trails.
We also run a monthly coherence check — a small sample of high-usage chunks gets reviewed against their source documents to catch cases where the source changed but the deprecation hook misfired. It's manual and takes about two hours. We haven't automated it yet because the failure rate is low enough that the overhead isn't worth it. That might change as the corpus grows.
If you want to see how we approach ongoing technical work across projects, the blog has more on this.
Key Takeaways
- Semantic recall means storing meaning as vectors, not storing text as text — so retrieval works even when the words don't match
- The chunking strategy matters more than the embedding model: 512 tokens with 64-token overlap is a solid starting point
- Feedback loops don't need to be real-time — a weekly batch job is more reliable and the quality difference is invisible to users
- We got one client from 6 minutes per query down to 90 seconds by letting the system carry context forward across sessions
- Don't build this if your users don't have continuity between sessions — a simpler RAG pipeline will serve them better and cost less to run
If you want Nuclear Marmalade to build something like this for your product, the fastest way to start is a conversation with our team. We'll tell you quickly whether the architecture fits — and what it'll actually cost to build.