MemVerse: Multimodal Memory and the Path Toward Lifelong Learning Agents

In building AI Agent systems, "memory" has always been the biggest barrier separating a simple language model from an intelligent entity capable of long-term companionship. Current solutions such as expanding the context window or using RAG (Retrieval-Augmented Generation) still suffer from fundamental limitations in computational cost and the ability to synthesize information over time.

This article analyzes MemVerse – a multimodal memory framework for Lifelong Learning Agents, released as an open-source preprint in December 2025. It represents a notable advancement by combining traditional retrieval mechanisms with real-time model weight updates.

The research is titled "MemVerse: Multimodal Memory for Lifelong Learning Agents", conducted by Junming Liu, Yifei Sun, and collaborators at Shanghai Artificial Intelligence Laboratory (Shanghai AI Lab). A key strength of this work is that the source code has been fully open-sourced, allowing the technical community to access and experiment with it at: https://github.com/KnowledgeXLab/MemVerse

The core focus of the research is addressing the problem of "Catastrophic Forgetting" and improving multimodal reasoning for agents operating in continuous interactive environments, rather than handling isolated tasks as in traditional approaches.

1. The Practical Problem

Currently, most AI Agents handle memory in two main ways:

1. Parametric Memory: Knowledge is stored within the model’s weights. The drawback is that it is static, difficult to update, and prone to Catastrophic Forgetting when continuously fine-tuned.
2. Non-parametric Memory: Uses a Vector Database for RAG. The drawback is lack of structure, slow retrieval at scale, and it only performs lookup rather than truly learning knowledge.

MemVerse addresses this problem by not choosing between the two, but instead combining both within a unified architecture, allowing the Agent to achieve both precise retrieval and fast intuitive responses.

2. Conceptual Foundation

The architecture of MemVerse is built upon Dual-Process Theory in cognitive science, most notably the System 1 / System 2 model proposed by Daniel Kahneman in Thinking, Fast and Slow (2011):

• Slow Thinking (Slow Pathway): Corresponds to retrieval-based long-term memory. It ensures high accuracy, detailed storage, and structured representation, but operates at a slower speed.
• Fast Thinking (Fast Pathway): Corresponds to parametric memory. This is a small language model (Small LM) trained to instantly respond to familiar information without complex retrieval.

The key idea is transformation: raw experiential data is stored in the slow memory, then periodically distilled into the fast memory.

3. Technical Analysis

Figure 1. Overall architecture of MemVerse. The system combines Short-term Memory for maintaining local context, Long-term Memory in the form of a Multimodal Knowledge Graph for structured storage, and Parametric Memory for fast retrieval via periodic fine-tuning. The Memory Orchestrator coordinates the entire storage and retrieval pipeline. Source: Liu et al., 2025.

The MemVerse system operates through a Memory Orchestrator that manages data flow across the following components:

Multimodal Processing

The agent does not directly store raw data (images, videos, audio) in a simple vector store. Instead:

• It uses VLMs (such as GPT-4o, BLIP) or audio models to convert input signals into textual descriptions.
• These descriptions are grounded back to the original files to ensure verifiability when retrieving evidence.

Long-Term Memory (LTM) – Knowledge Graph

Unlike standard RAG systems that rely on flat vector search, MemVerse organizes memory as a Hierarchical Knowledge Graph, consisting of three layers:

1. Core Memory: Stores essential user information and personal preferences.
2. Episodic Memory: Stores sequences of events over time.
3. Semantic Memory: Stores generalized knowledge abstracted from specific events.

This graph structure enables multi-hop reasoning, which similarity search in vector databases often fails to achieve.

Parametric Memory – Distillation Mechanism

This is the most innovative component of MemVerse. To reduce latency and increase responsiveness:

• The system periodically extracts question–answer (QA) pairs from the Knowledge Graph.
• These pairs are used to perform Supervised Fine-tuning (SFT) on a small language model (e.g., Qwen-7B).
• As a result, important knowledge is embedded directly into the model’s weights. When encountering similar queries, the agent responds immediately using Parametric Memory without needing to query the graph.

The weight update process is described as a continuous accumulation:

$$\mathcal{M}^{t+1}_{parametric} = \mathcal{M}^{t}_{parametric} + \Delta \Theta_t$$

• $\mathcal{M}^{t}_{\text{parametric}}$ represents the state of parametric memory at time (t), i.e., the full set of weights of the small language model after being trained on prior experiences.
• $\Delta \Theta_t$ denotes the parameter update obtained from a new round of Supervised Fine-tuning, based on data pairs extracted from Long-term Memory (typically question–retrieved answer pairs).

Figure 2. Illustration of multimodal memory and grounded reasoning in MemVerse. Compared to models without memory, MemVerse enables agents to retrieve multimodal evidence, reduce hallucinations, and maintain long-term context. Source: Liu et al., 2025.

4. Practical Perspective

From a system implementation perspective, this architecture presents the following evaluation:

Advantages:

• High performance: The use of Parametric Memory significantly reduces latency for repeated queries. The paper reports ~72% faster retrieval compared to using Long-term retrieval alone.
• Scalability: The architecture is model-agnostic, allowing flexible replacement of LLM/VLM modules depending on hardware resources.
• Sustainability: It effectively mitigates catastrophic forgetting by using the Knowledge Graph as a persistent storage backbone, while still enabling continuous learning.

Challenges:

• Infrastructure cost: Maintaining periodic fine-tuning requires stable GPU resources and is more expensive than pure RAG systems.
• Graph construction complexity: Extracting entities and relations from unstructured conversational text remains a challenging problem and is prone to noise, affecting graph quality.
• Synchronization: A robust mechanism is needed to avoid inconsistencies between parametric memory (in model weights) and external memory (in the graph) when information changes.

5. Conclusion

MemVerse is not merely a storage technique, but a shift in paradigm from Static RAG to Dynamic Memory Systems. Combining the storage power of Knowledge Graphs with the responsiveness of fine-tuned small language models is a promising direction for applications such as AI companions, game NPCs, and advanced virtual assistants.

For engineers building AI systems, studying the "dual-pathway" mechanism of MemVerse can provide valuable insights into balancing cost, speed, and memory capability in modern AI architectures.