How Memory Really Shapes Human Reward Learning

For decades, reinforcement learning (RL) has been the dominant framework for explaining how humans learn from rewards. At its core, RL assumes that people maintain a small set of internal variables-most notably Q-values-that summarize past outcomes and guide future decisions through simple, incremental updates.

This assumption has powered progress across psychology, neuroscience, and AI. But as datasets have grown and behavioral analysis has become more precise, cracks in this picture have become increasingly difficult to ignore.

Where Classic Reinforcement Learning Breaks Down

Human behavior routinely violates the assumptions of classic RL. Individual experiences can exert long-lasting influence, learning is sensitive to global reward context, and neural signals associated with value appear far richer than a single scalar estimate can explain.

These observations suggest that humans do not rely solely on compressed reward summaries. Instead, learning appears to involve multiple internal memory systems operating over different timescales.

Why Neural Networks Alone Are Not the Answer

Recurrent neural networks (RNNs) offer an obvious alternative. They can store high-dimensional internal states and capture long-term dependencies across sequences of events. When trained to predict human behavior, generic RNNs often outperform traditional RL models.

However, this performance comes at a cost. Pure neural models behave as black boxes, offering little insight into what cognitive mechanisms they represent or why they succeed.

Hybrid Neural–Cognitive Models

To reconcile predictive power with interpretability, researchers introduced hybrid neural–cognitive models. These models preserve the architectural structure of cognitive theories while replacing rigid, hand-crafted update rules with flexible neural components.

Rather than assuming a specific learning equation, each component is allowed to learn its own function from data-within constraints inspired by cognitive theory.

Testing Learning at Real Scale

The models were evaluated on a large-scale human dataset collected from a non-stationary bandit task. Participants repeatedly chose between four options whose reward distributions drifted over time.

The dataset included hundreds of participants and hundreds of thousands of trials, enabling rigorous comparison without overfitting-a critical requirement when evaluating flexible models.

Why More Expressive RL Still Fails

The strongest handcrafted RL baseline incorporated extensions such as perseveration, forgetting, and variable learning rates. A hybrid version replaced its update equations with neural networks, allowing it to learn any nonlinear variant of RL-style learning.

Despite this flexibility, the model converged on update rules that closely resembled classic RL and failed to significantly improve predictive accuracy. Simply making RL more expressive was not enough.

Context Matters-but It’s Still Not Enough

Humans evaluate outcomes relative to context, not in isolation. A context-aware hybrid model allowed learning updates to depend on the values and histories of unchosen actions, substantially improving predictions.

Even so, this model still fell short of generic RNN performance, indicating that something deeper was missing.

Memory Is the Missing Ingredient

The breakthrough came with a model that explicitly separated memory from decision variables. Instead of treating value estimates as memory, the model introduced dedicated recurrent memory states capable of retaining rich representations of past experience.

These memory states tracked information across multiple timescales, from recent outcomes to long-term environmental trends, enabling the model to match the predictive accuracy of a full RNN while remaining interpretable.

What These Memory States Encode

Analysis revealed that different dimensions of memory integrated reward information differently. Some responded strongly to recent events, while others captured slow-moving shifts in reward context.

Crucially, certain memory components correlated with response times-despite timing never being part of the training objective-indicating that the model had uncovered latent cognitive variables.

Implications for AI and Cognitive Modeling

These results challenge the idea that learning is driven solely by incrementally updated scalar values. Instead, behavior emerges from structured memory systems that preserve rich information about the past.

For AI builders, this suggests that separating memory from value estimation may be critical for building agents that adapt robustly in non-stationary environments.

Why Founders Should Care

If you are building autonomous agents, recommendation systems, or adaptive decision engines, this work highlights a core limitation of value-centric learning. Systems that compress experience too aggressively lose critical signal.

Architectures that preserve memory-explicitly and flexibly-are better positioned to handle drift, context shifts, and long-horizon behavior.

Closing Thoughts

Reinforcement learning remains a powerful abstraction, but it is not a complete theory of learning. By combining neural flexibility with cognitive structure, hybrid models reveal where RL works, where it fails, and what must replace it.

Human learning is shaped not just by reward, but by memory. Understanding that memory-how it is structured, updated, and used-may be the key to building AI systems that truly learn.