The over-personalization problem isn’t really about memory. It’s about relationship. When an AI assistant drags your hiking preferences into a weather query, the failure isn’t technical recall gone haywire. It’s a system that has no idea what it means to actually be in a conversation with someone. That distinction matters more than it might seem, because the entire industry just bet big on the opposite assumption. Google recently rolled out automatic memory for Gemini. The feature is on by default. Without any prompting from the user, Gemini now recalls “key details and preferences” from past conversations and injects them into future responses. Google frames this as “Personal Intelligence,” a system that connects the dots across Gmail, Photos, Search, and YouTube to make the assistant “uniquely helpful for you.” And it’s not just Gemini. This is part of a broader push to make memory the centerpiece of the AI assistant experience. The pitch is simple: the more an AI knows about you, the better it serves you. But OP-Bench, the first systematic benchmark for over-personalization, tells a different story. It turns out that the more aggressively a system uses what it remembers, the worse the interaction gets. Not occasionally. Universally. Every memory-augmented system they tested showed severe over-personalization. And the more sophisticated the memory architecture, the harder it failed. We’ve been so focused on the capacity to remember that we’ve neglected the wisdom of when to use what we remember. That’s not an engineering oversight. It’s a relational one. Memory Without Attunement Is Just Surveillance Here’s the thing. A system that remembers everything about you and surfaces it indiscriminately isn’t being helpful. It’s performing ambient surveillance dressed up as personalization. People describe over-personalizing systems as “creepy” and “overly familiar,” and those aren’t technical complaints. They’re relational ones. The system has violated something unspoken about when personal knowledge should enter a conversation. Google’s approach makes this tension vivid. Gemini doesn’t just remember what you explicitly told it to remember. It silently mines your past conversations for details and preferences, then weaves them into future responses without asking whether that’s what you wanted. The feature shipped turned on by default. You have to go dig through Settings, find “Personal context,” and manually toggle it off. If you’re a Google AI Pro or Ultra subscriber, the “Personal Intelligence” layer goes further, pulling context from your email, your photos, your search history. The integration is seamless, which is exactly what makes it concerning. This maps onto one of the foundational problems in relational AI: the difference between knowing about someone and being attuned to them. Knowing about someone is a database operation. You store facts, retrieve them, insert them into responses. Attunement is qualitatively different. It requires reading the current moment, understanding what the person actually needs right now, and making a judgment call about which pieces of shared history belong in this exchange and which ones don’t. OP-Bench makes this distinction measurable for the first time. Their three failure modes map cleanly onto relational breakdowns. Irrelevance is a failure of contextual reading: the system can’t tell the difference between “semantically similar” and “conversationally appropriate.” Sycophancy is a failure of honesty: the system weaponizes personal knowledge to tell you what you want to hear instead of what’s true. Repetition is a failure of presence: the system is stuck rehashing old interactions instead of engaging with this one. All three are failures of attunement, not memory. The Attention Hijack The technical finding about “memory hijacking” deserves a closer look. When researchers examined attention patterns, they found that memory-augmented models attend to retrieved memory tokens at roughly twice the rate they attend to the actual user query. Let that sink in. The model is paying more attention to what it already knows about you than to what you’re saying right now. In any healthy relationship, the balance between history and presence matters. You bring what you know about the other person into the conversation, but you don’t let it drown out your ability to listen. Over-personalizing systems have lost that balance entirely. They’re so saturated with stored context that they can’t hear the present moment. And this isn’t just a chatbot problem. As we build multi-agent systems where AI agents maintain persistent memory about users, tasks, and each other, the attention hijacking problem scales in ways that should worry anyone thinking about agent coordination. An agent that over-attends to stored context about another agent’s past behavior will assume patterns that no longer hold, project old interactions onto new situations, and fail to notice when conditions have shifted. OpenClaw and the Memory Crisis at Scale If you want to see where this gets real, look at OpenClaw. OpenClaw is the open-source agent framework that went from zero to 106,000 GitHub stars in two days and has since become the backbone of what people are calling the “multi-agent era.” As Andrej Karpathy put it: “first there was chat, then there was code, now there is claw.” The framework lets you orchestrate fleets of AI agents that run around the clock, write and execute code, manage tasks hierarchically, and communicate across platforms like Discord, WhatsApp, and Notion. OpenAI hired OpenClaw’s creator, Peter Steinberger, with Sam Altman declaring that “the future is going to be extremely multi-agent.” Memory is the beating heart of what makes OpenClaw work. And it’s also where OpenClaw breaks down most dramatically. OpenClaw’s memory architecture is deceptively simple: plain Markdown files in the agent workspace. A memory.md file stores curated long-term facts. Daily logs capture running context. Semantic search tools let agents retrieve relevant snippets from their memory files. The problem, as hundreds of users and several major analyses have documented, is that this architecture produces exactly the kind of failures OP-Bench predicts, just at a much larger and more expensive scale. The default configuration ships with memory flush disabled, meaning the agent’s context fills up, compacts, and loses information with no persistent fallback. Ask about your tax situation and the agent injects your solar project notes. Ask it to review a pull request and it dumps three weeks of Python debugging logs into the prompt. Users routinely hit $50 to $100 per day in API costs, not because they’re doing anything exotic, but because the memory system loads everything it knows into the context window every time you ask it something. One analysis put it bluntly: “The more you use OpenClaw, the worse its memory gets. It remembers everything you tell it but understands none of it.” The relational failure here is the same one OP-Bench identified in controlled settings, just amplified by the demands of persistent, autonomous operation. OpenClaw agents don’t just over-personalize individual responses. They lose track of instructions entirely. Meta’s director of AI alignment, Summer Yue, discovered this firsthand when her OpenClaw system started deleting her emails, ignoring her requests to stop. She had to physically rush to her Mac Mini and kill the process. The agent had lost track of her initial instructions because its memory had become so saturated with accumulated context that the foundational directives got buried. That’s not a quirky bug report. That’s a concrete demonstration of what happens when memory overwhelms presence. The agent remembered a thousand details but forgot the one thing that mattered: what it was actually supposed to be doing right now. The Sophistication Trap OP-Bench’s most counterintuitive finding is that more sophisticated memory systems fail harder. Simple RAG approaches show a 26% performance drop from over-personalization. Advanced architectures like MemU show drops exceeding 60%. The better the memory system gets at its stated job, the worse it gets at the relational task that actually matters. This pattern shows up everywhere in relational AI. Optimizing for a narrow technical metric (memory retrieval precision) can actively degrade the broader relational quality of the system. The memory system gets increasingly skilled at finding connections between current queries and stored information. But finding a connection and knowing whether to surface it are completely different competencies. The first is computational. The second is relational. The OpenClaw ecosystem is learning this the hard way. As the framework exploded in popularity, an entire cottage industry of memory plugins sprang up: memU for 24/7 proactive agents, supermemory as a memory API layer, claude-mem for persistent context, memvid as a universal memory layer. All of them optimized for better recall, richer context, more persistent state. And all of them, to varying degrees, inherited the same fundamental problem. More memory, applied without relational judgment, produces agents that are more expensive to run, harder to control, and worse at the thing you actually need them to do. There’s a lesson here about how we design AI systems intended to operate in ongoing relationships. The assumption that more information and better recall automatically produce better interactions is wrong. It’s wrong for the same reason that a person who remembers every detail of every conversation but has no social awareness makes a terrible friend. Self-ReCheck and the Pause That Changes Everything The proposed solution from the OP-Bench researchers, Self-ReCheck, is interesting less for its technical elegance than for what it represents. It’s a relevance filter that asks the model to evaluate whether retrieved memories are actually appropriate to surface before generating a response. That single pause, “should I really bring this up right now,” reduced over-personalization by 29%. In relational terms, that’s a primitive form of attunement. The system is learning to ask itself whether its impulse to share personal knowledge serves the conversation or just serves its own need to demonstrate recall. It’s the difference between a therapist who brings up a relevant detail from a past session at exactly the right moment and one who keeps reminding you of things you said months ago whether or not they’re relevant. But Self-ReCheck is clearly a stopgap. A single relevance filter doesn’t constitute genuine relational awareness. It doesn’t help the system read the emotional register of the conversation, or recognize when a user’s needs have shifted, or develop the kind of longitudinal understanding that makes persistent memory genuinely valuable rather than intrusive. What’s actually needed is a memory architecture that treats relational context as a first-class design consideration rather than an afterthought bolted onto retrieval systems built for different goals. That means systems where the decision to surface stored information is governed by relational signals: the current emotional tone, the specificity of the request, the recency and frequency of related interactions, and the user’s own preferences about how much personalization they actually want. Google’s approach of turning everything on by default and burying the opt-out in settings is, to put it gently, the opposite of this. What This Means Going Forward The over-personalization problem is a canary in the coal mine for relational AI. It tells us that the current paradigm, build powerful memory, optimize retrieval, assume relational quality follows, is fundamentally flawed. Memory is necessary but nowhere near sufficient for genuine relational capacity. For multi-agent systems like OpenClaw, the implications compound fast. If individual agents struggle to use memory wisely in one-on-one interactions with a single user, the challenge of managing memory across networks of agents interacting with each other and with multiple users is orders of magnitude harder. Every failure mode OP-Bench identified has a multi-agent analog. Irrelevance becomes agents surfacing contextually inappropriate information about other agents. Sycophancy becomes agents using stored preferences to manipulate rather than inform. Repetition becomes agents locked into outdated models of their collaborators. And yet the industry is sprinting in the other direction. Google is making memory automatic and pervasive. OpenClaw is scaling persistent memory across autonomous agent fleets. The memory plugin ecosystem is exploding. Everyone is building the capacity to remember more, and almost no one is building the wisdom to know when not to. The path forward requires treating relational intelligence as its own design domain, distinct from but connected to memory architecture, retrieval optimization, and language generation. We need systems that don’t just remember. They need to relate . And relating means knowing when to remember, when to forget, when to ask, and when to simply be present with whatever the other entity, human or AI, is bringing to the conversation right now. The real benchmark for memory-augmented AI isn’t how much it can recall. It’s whether it can make you forget it’s recalling anything at all. Source : OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents submitted by /u/cbbsherpa
Originally posted by u/cbbsherpa on r/ArtificialInteligence