LongMemEvalMethod & sources ↓Evidence Recall@10 (%) · 0–100; higher is betterEarlier consolidated report · retrieval evidence, not answer accuracy
95.2
—
—
—
—
—
—
86.2
LoCoMo-PlusMethod & sources ↓Factual average · 0–100; higher is betterEarlier consolidated report · paper references and company result
61.52
57.24
—
59.64
57.53
—
—
—
MemBenchMethod & sources ↓Participation Scenario · Accuracy · 0–1; higher is betterEarlier consolidated report · paper references and local baseline
0.892
—
—
—
—
0.455
0.722
—
Experimental results are provided for reference only and do not constitute academic research. Results may be updated at any time as new evaluations become available.
Benchmark methods & resources
Open a benchmark for its original resources, procedure, selected metric and reported-run scope. These descriptions explain the public protocols; they are not a claim that every company run reproduced the current upstream configuration.
Tests whether an assistant can answer questions using information distributed across a long conversation history.
Dataset and selected task
The public release contains ten multi-session conversations with timestamps, QA labels and evidence dialogue IDs.
Evaluation procedure
Ingest conversation history, retrieve relevant memory for each question, generate an answer, then compare it with the annotated answer using the QA evaluator.
Metric and calculation
This row reports Overall Token-level F1 on a 0–1 scale: token precision and recall measure overlap with the reference answer. Higher is better; no LLM judge is needed for this selected metric.
Scope of the reported run
All three populated cells come from the latest company report. Sample selection, category inclusion, answer normalization and the exact Overall aggregation are not supplied; the repository’s full dataset size does not establish the company run size.
Score provenance
CoSens: Company run · Mem0: Company run · TencentDB Agent Memory: Company run
Tests hallucinations in memory extraction, memory updating and downstream question answering as separate operations.
Dataset and selected task
HaluMem offers Medium and Long user-centric conversation histories, annotated memory points and QA pairs. This row selects Medium QA only.
Evaluation procedure
Feed sessions through a memory-system adapter, collect stored and updated memories and QA predictions, then run the evaluation suite against annotated reference information.
Metric and calculation
Medium QA Correct is the reported correctness measure on a 0–1 scale. The official suite also evaluates memory extraction and update accuracy, hallucinations and omissions; those are not averaged into this row.
Scope of the reported run
The three values are company-run results in the latest report. Judge version, prompts, thresholds and evaluated QA count are not recorded. A high QA value alone does not establish extraction or update leadership.
Score provenance
CoSens: Company run · Mem0: Company run · TencentDB Agent Memory: Company run
Tests memory through incremental interactions, including accurate retrieval, test-time learning, long-range understanding and conflict resolution.
Dataset and selected task
The selected Conflict Resolution task provides facts that change or conflict across incoming context chunks.
Evaluation procedure
Process chunks into memory in sequence, then answer questions that require the applicable updated fact rather than an outdated conflicting version.
Metric and calculation
MemoryAgentBench uses an adapted Token F1 in this company report, on a 0–1 scale. The official repository describes substring exact-match accuracy for its conflict-resolution tasks. These scoring protocols are different.
Scope of the reported run
Only Mem0 OSS and TencentDB have reported values; the CoSens result is missing. The adaptation code, chunk size, task revision, sample count and normalization rules are not supplied, so these values are not official-protocol accuracy.
Score provenance
Mem0: Company run · TencentDB Agent Memory: Company run
A synthetic organization with multiple authors, channels and valid-time versus ingestion-time history. The public Medium tier has 443 artifacts and 73 questions.
Evaluation procedure
Ingest the selected tier with a system adapter, retrieve evidence, generate answers and judge them against structured rubric facets. The harness supports both native and shared answerers.
Metric and calculation
Overall rubric score is mean rubric-weighted facet coverage in [0,1], allowing partial credit. This row keeps the company report’s Medium result; its Small result is not mixed in.
Scope of the reported run
Only the CoSens company-run value is reported here. The public tier size describes the benchmark; actual evaluated question IDs, corpus revision, answerer, judge and run count are not supplied by the company report.
Tests long-term interactive memory, including extraction, multi-session and temporal reasoning, knowledge updates and abstention.
Dataset and selected task
Public S, M and oracle variants each contain 500 instances, with evidence session IDs and answer-bearing turn annotations. The company report does not identify its exact variant.
Evaluation procedure
For retrieval evaluation, index the history, retrieve the ten highest-ranked memory items for each question, and check against annotated evidence. The benchmark separately supports generated-answer evaluation.
Metric and calculation
Evidence Recall@10 (%) measures reference evidence recovered in the top ten results, on a 0–100 scale. It is not end-to-end answer accuracy. The source does not provide the exact matching and aggregation implementation.
Scope of the reported run
CoSens is a company-run result; Session RAG (V+fact) is a Table 3 paper reference. Retrieval unit, evidence matching, embedding version, query expansion and sample selection are not established as matched.
Score provenance
CoSens: Company run · Session RAG (V+fact): Paper reference
Adds cognitive memory to long conversational QA: use an earlier cue to respect a later implicit constraint, even when their wording differs.
Dataset and selected task
Extends LoCoMo’s factual categories with cue–trigger cognitive tasks. This table selects the reported factual average, not the cognitive score or a combined total.
Evaluation procedure
Build unified inputs, obtain predictions from the model or memory system, then score with the released LLM-as-judge framework.
Metric and calculation
The released judge awards correct, partial and wrong responses 1, 0.5 and 0. This row preserves the report’s factual average on a 0–100 scale; the company aggregation implementation is not supplied.
Scope of the reported run
CoSens is a company-run value. Mem0, A-Mem and SeCom are paper references copied into the consolidated report. Model, prompt, judge and exact sample equivalence have not been established.
Score provenance
CoSens: Company run · Mem0: Paper reference · A-Mem: Paper reference · SeCom: Paper reference
Paper ↗https://aclanthology.org/2025.findings-acl.989/
What it tests
Tests factual and reflective agent memory in participation and observation scenarios, with effectiveness, efficiency and capacity dimensions.
Dataset and selected task
Participation data represents first-person interaction; observation data represents third-person information. Released data supports different conversation lengths and added distractor content.
Evaluation procedure
Feed the chosen interaction history through a memory method, answer the associated questions and score correctness for the selected scenario and context-length condition.
Metric and calculation
This row uses reported Participation Scenario Accuracy on a 0–1 scale. Observation accuracy, efficiency and capacity are separate measures and are not combined into this value.
Scope of the reported run
CoSens and RAG (local) are company-run results; MemGPT is a paper reference. The report describes its local RAG comparison as matched, but exact lengths, question IDs, sample counts and scoring settings are not supplied.
Score provenance
CoSens: Company run · MemGPT: Paper reference · RAG (local): Local baseline
Exact evaluated versions are not provided in the source reports. Project links identify the original systems; a current release may differ from the implementation behind a historical score.
CoSens
Experience layer with persistent memory and reusable methods. The reported memory implementation is named Friday in the source reports.
Latest-report rows use company-run Mem0 results; MemoryAgentBench explicitly uses Mem0 OSS. LoCoMo-Plus uses a paper reference. Platform, OSS and graph variants must not be treated as interchangeable.
TencentDB Agent Memory
TencentDB Agent Memory provides persistent memory infrastructure for agents.
Company-run results in the latest report. The exact evaluated release and storage configuration are not recorded; the current repository may differ from the evaluated version.
A-Mem
Agentic memory that organizes notes, metadata and links, and updates relationships as new memories arrive.
LoCoMo-Plus values are paper references reproduced in the company report. The system repository distinguishes the memory implementation from the authors’ experiment code.
SeCom
Conversation memory built from topical segments, with compression and retrieval over those memory units.
MemBench uses the historical MemGPT reference from the source report. The current Letta project is its successor, not proof that this score measures today’s Letta release.
RAG (local)
The local retrieval-augmented memory baseline labeled ragmemory in the consolidated report.
A company-run baseline, not a separate publicly identified product. The report describes matched local conditions for its MemBench comparison but supplies no public implementation URL or complete configuration.
Session RAG (V+fact)
A LongMemEval paper reference using sessions as retrieval values, with session text and extracted facts as retrieval keys (K=V+fact).
The 86.2 value is the paper’s Table 3 retrieval reference transcribed into the company report. It is a method configuration, not an independently released memory product.
Sources & run settings
Reviewed 2026-10-17. Higher is better within each row. Scores retain their original scales; unrelated tasks are not averaged into a composite total. A dash means no reported result.
Latest Memory comparison table
Source of the LoCoMo, HaluMem, MemoryAgentBench and OrgMemBench scores. Reviewed 17 October 2026.
Source of the LongMemEval, LoCoMo-Plus and MemBench rows. Combines company results, published references and a local RAG baseline. Reviewed 17 October 2026.
The CoSens column uses the memory implementation called Friday in the reports. These results do not benchmark the complete SDK or the Learn loop. Results come from multiple source reports. Older rows include paper-reported references and company runs; matching models, samples and settings are not established. Compare only within the stated metric and subset; these are not a controlled cross-system reproduction.
Available reproduction information
The fields below describe the company-run evidence available for this table. Missing settings remain explicit; upstream quickstarts are linked above and should not be read as the commands used for these runs.
Dataset revision and split Not provided
No pinned dataset commit, checksum or complete split declaration accompanies the reported runs.
Sample IDs and counts Not provided
Public dataset sizes are described below; actual run manifests and evaluated counts are not supplied.
Memory implementation versions Not provided
No evaluated commit, package version or deployment revision is supplied for the company runs.
Generator and embedding models Not provided
No complete per-system model versions, embedding configuration or provider settings are supplied.
Retrieval and ingestion settings Partly provided
LongMemEval selects top 10; other chunking, top-k, filtering, memory-write and update settings are not supplied.
Judge, prompts and aggregation Partly provided
Selected metrics are named. Exact company evaluator revisions, prompts, normalization and aggregation code are not supplied.
Run dates, repetitions and uncertainty Not provided
Report review date is not a run date. Seeds, repeat counts and confidence intervals are not supplied.
Raw predictions and run artifacts Not provided
This page publishes transcribed scores and references; raw per-question outputs, execution logs and reproduction commands for the company runs are not supplied.