Evaluating Very Long-Term Conversational Memory of LLM Agents
Abstract
Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very long-term dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering (2024)
- ChatQA: Building GPT-4 Level Conversational QA Models (2024)
- Commonsense-augmented Memory Construction and Management in Long-term Conversations via Context-aware Persona Refinement (2024)
- CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation (2024)
- Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Really awesome effort @adymaharana and team! Would you consider putting your dataset on the hub? @davanstrien can help with this!
Hi @derek-thomas , thanks for the suggestion! Yes, we have been contemplating putting up the dataset on HF as well. I have experience with this, so I can manage. Will update this page with a link to the dataset when it's up, thanks!
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper