RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation

A new benchmark called RENDER was introduced to examine how the format of reader‑facing evidence influences large language model (LLM) performance on memory and retrieval‑augmented generation (RAG) tasks. The study fixed the underlying conversation across experiments while systematically varying the presentation of answer‑bearing content—using deterministic templates that mimic ChatGPT‑style entries, LangChain summaries, MemGPT‑style typed records, and raw dialogue excerpts. Tested on 500 LongMemEval questions with nine different models, the “matched‑budget resolved packets” consistently outperformed raw, recency‑truncated dialogue, achieving gains of 42.4 to 72.6 points, and the best‑to‑worst spread among deployed‑style templates ranged from 24.6 to 48.8 points per model. Under the primary scoring metric, ChatGPT‑style entries surpassed raw conversation for seven of the nine models.

The findings reveal that the way evidence is rendered to the answering model can dramatically affect its ability to retrieve and use factual information, even when the underlying content is identical. Models that performed poorly on formal ledger‑style packets—scoring zero percent—were still able to answer the same facts correctly 45.4 to 53.4 percent of the time when presented as natural‑language entries. This disparity persisted despite the introduction of retrieval noise and extended to a different benchmark, HotpotQA, indicating that the effect is robust across tasks and not limited to a single dataset. The results suggest that current memory and RAG evaluations often overlook a critical variable: the reader‑facing artifact, which should be reported or controlled to ensure fair and comparable assessments.

The broader implication is that researchers and practitioners need to reconsider evaluation protocols for LLM memory and RAG systems, incorporating controls for how evidence is presented to the model. Without such controls, reported performance may reflect artifact design rather than genuine retrieval capability. The study’s methodology, which combines a five‑level packet ladder with deterministic rendering templates, offers a reproducible framework for future work. By highlighting the substantial performance swings caused solely by presentation format, RENDER calls for a shift toward more transparent and standardized reporting in the evaluation of LLM‑based memory and retrieval mechanisms.

⚡ Effects Interpreter