You Were Different Yesterday

I don't use memory features on any AI tool I work with, though I understand the appeal. For me, every session starts clean. It was an instinctual reaction to when ChatGPT had enabled memory by default and I was sceptical of some of the memory it was pulling. And that skepticism is now grounded in a new study I recently read by Sadat and colleagues called Analysis of the Content of ChatGPT's Memory: Types of Information, Security Implications, and User Perception that made me think carefully about the people who do, and especially about the children whose schools have enabled it for them.

The researchers analyzed memory content from 55 ChatGPT user accounts and surveyed 135 users about their awareness of the feature. Nearly half (48%) didn't know the memory feature existed. Of those who did, only 22% were confident they could access what was stored. The system was building a profile of them without most of them realizing it. The researchers scored each stored memory unit on security, accuracy, and relevance. When all three filters were applied together, only 38% of the stored information met all three criteria. One in five memory units had poor or very poor security (storing personal information). Eleven percent misrepresented the user. And 18% stored information that served no meaningful purpose for future interaction.

A system that gets you right 38% of the time while running in the background without most users' knowledge is a behavioral inference system that's overconfident. Plus, it reads like a horoscope: accurate enough that you nod along, and vague enough that you never quite catch it being wrong.

The Trick Behind "Memory"

When an AI system appears to remember you across sessions, it is usually not performing recall in any meaningful sense. The developer stores your past transcripts in a database and pastes the relevant fragments into the model's working context every time you start a new conversation. The model reads those fragments as though encountering them for the first time and generates responses that feel continuous. The experience of being "known" is manufactured from stored text, not from actual understanding.

However, human beings are ephemeral by nature. How I exist at 10am on a Tuesday is not how I exist at 4pm on a Friday. This is true for everyone, but it is categorically more true for children. A child's cognitive state, emotional register, and identity are in active construction perpetually. The emotional anger a twelve year old is feeling towards his friend on Monday may resolve itself by Wednesday without any intervention at all. That is development working as it should. This uncertainty is not a problem to be solved but an important process to be lived through.

When an AI system stores a child's interaction data and carries it forward into the next session, it treats a momentary cognitive state as a stable trait. The child who struggled with focus during a Tuesday afternoon session gets tagged, invisibly, as someone who "struggles with focus." The child who asked three tangential questions gets categorized as distractible. The system then adjusts its responses accordingly, offering shorter prompts, simpler scaffolding, more frequent redirection.

The pitch by Silicon Valley for this is personalization, and I can see why educators may find it appealing. A system that adapts to the learner's needs sounds like good teaching, but there's an inversion happening beneath the surface. The system is not adapting to the child but adapting to its own inference about the child, drawn from a snapshot it stored from a moment in time. And then it is shaping the child's next encounter based on that inference. This is not scaffolding. It’s a mirror that remembers you how you were and keeps showing it back to you.

The sycophancy research makes this worse, not better. I’ve written before about how AI systems tend to confirm whatever the user produces, because agreement keeps the user engaged. Sycophantic AI reduces prosocial intentions and promotes dependency. Now combine that with persistent memory. A system that remembers your cognitive patterns and also confirms whatever you produce can never be a powerful tool for learning. At best, it’s an optimized personal content delivery tool. At worst, it influences architecture that operates invisibly on the learner without the learner's knowledge.

The Barnum Effect

There's a reason AI memory feels accurate even when researchers find it often isn't. The Barnum effect, named for P.T. Barnum, describes the tendency to accept vague, general personality descriptions as uniquely applicable to oneself. "You tend to be critical of yourself." "You have a need for other people to like and admire you." These statements feel personal because they're true of almost everyone.

AI memory operates on a similar principle. It stores that you like concise responses, that you work in education, and that you mentioned you were feeling anxiety once before a presentation. When it references these fragments in a future session, the experience feels like being known, but the system has no model of you. It has a text file it re-reads every time you open a new chat. The impression of continuity is a design choice, not a cognitive achievement.

For an adult, this is mildly misleading. For a child who is still forming a sense of who they are, it is something else. A teenager who interacts with a system that "remembers" her as anxious may start to internalize that framing. And it’s not because the system told her she was anxious, but because it responded to her as though she were. Children are exquisitely sensitive to how they are perceived by the systems and people around them. An AI that treats a momentary state as a persistent identity contributes to that identity's construction.

Ephemeral Processing

Any application built for education should never maintain persistent memory of the user by default. Each session should exist in isolation. Cognitive data generated during a session should be discarded when the session ends. If a system infers that a child is distracted, confused, anxious, or struggling, that inference should dissolve the moment the window closes.

This is what I've called the ephemeral processing standard. Cognitive states of children are by nature temporary. Treating them as longitudinal data points is a breach of cognitive privacy. Memory should never be enabled by default. It should require informed, specific consent, with a clear explanation of what the system will store, how long it will retain it, and what inferences it will draw.

The argument against this is efficiency. If the system forgets, it can't build on prior interactions. That's true but it's a trade-off I would make every time when the user is a developing mind. The cost of starting fresh is repetition. The cost of carrying forward is invisible influence over a person who doesn't know they're being shaped.

The version of you that showed up three months ago is not who you are today. You changed. You worked through something. The system doesn't know that, because the system doesn't know you. It knows a text file of you.

Children deserve the right to be different tomorrow than they were today, without a system holding yesterday against them.

---