HIPPOGIST: Dual-Store Memory for Long-Horizon LLM Agents with Visual Traces and Semantic Gists

Shutong Liu

Beihang University

HIPPOGIST architecture: chunk-by-chunk reasoning, visual and textual memory updates, and outcome-supervised DAPO training. — **Remember the meaning. Retain the evidence.** A shared Qwen3-VL-8B policy reads a document in chunks, maintaining both visual source traces and a compact semantic gist.

Abstract

Rendering long text as images offers a route to context compression, but a smaller token count does not by itself ensure reliable reading. Our analysis examines repetitive generation, confusion between similar passages, and concentrated or diffuse attention during visual reading. Meanwhile, a running textual summary can discard evidence needed later. We introduce HIPPOGIST, a dual-store architecture that reads a long document chunk by chunk while maintaining compressed visual source traces and a short textual semantic gist. Optional image tools support inspection of the active input. We formulate training of the shared Qwen3-VL-8B policy with a multi-conversation DAPO objective: the final-answer reward supplies an advantage to generated tokens in every preceding round, including memory updates. LongBench v2 experiments examine the trade-offs between separate textual-memory and optical-memory workflows. In a HotpotQA-trained optical-memory run, the score changes from 28.0 to 29.8 overall and from 25.9 to 32.4 on the Long subset between recorded steps 0 and 15. The reported scores evaluate these individual memory workflows.

Two Complementary Memories

A running summary is compact, but may omit details that become useful later. Visual compression retains another view of the source, but legibility and reliable evidence selection remain challenges. HIPPOGIST proposes to maintain both representations at every reading round.

Semantic gist

A short, explicit reasoning state with a proposed 1,024-token budget. It tracks the goal, supported facts, open questions, and recent context. The policy rewrites it after each chunk.

Visual traces

Rendered images of observed source content preserve access to wording and layout. A processor-aware visual-token budget controls retention, downsampling, and eviction.

At each round, the policy receives the question, the current chunk, and both memories. It may inspect active images with high_light or resize, then updates the gist. The visual store incorporates the newly observed source. After the final chunk, the policy answers from the retained memory.

Learning from the Final Outcome

HIPPOGIST formulates multi-conversation DAPO training over complete reading trajectories. A verifier scores the final answer, and that trajectory’s advantage weights eligible generated-token losses across every round, including earlier memory updates.

The aggregate loss updates the shared policy. Credit assignment operates through the policy-gradient objective; it does not differentiate through sampled text, image rendering, or resizing. Input memories and tool outputs are excluded from the generated-token loss.

Experiments

We study textual-memory and optical-memory workflows on LongBench v2. The results below evaluate the two memory representations separately, rather than the combined dual-store architecture.

LongBench v2 Overall and Long scores at recorded steps 0 and 15 for optical and textual memory. — **Outcome-supervised memory learning.** LongBench v2 accuracy at checkpoints 0 and 15 of the HotpotQA-trained optical-memory and textual-memory runs. Bars show aggregate scores for each checkpoint.

LongBench v2 accuracy (%)
Memory workflow	Recorded step	Overall	Long subset
Optical memory	0	28.0	25.9
Optical memory	15	29.8	32.4
Textual memory	0	35.4	35.2
Textual memory	15	36.2	36.1

The optical run improves by 1.8 percentage points overall and 6.5 points on Long. The textual run achieves higher absolute scores at both checkpoints.

Representation choices show trade-offs. In the Qwen2.5-VL experiments, an optical summary plus window images scores 29.4 overall and 26.9 on Long, compared with 29.8 and 20.4 for a 1,024-token text summary. These configurations differ in rendering and memory content; their scores illustrate representation trade-offs rather than an isolated component effect.

Memory Budget & Scope

Fixed gist and visual-token budgets bound active history. With bounded current input and tool calls, processing can proceed chunk by chunk while cumulative work grows with document length. Retention and rendering quality determine how much source evidence remains available to the policy.

Downsampling can destroy characters, eviction can remove evidence, and summaries can overwrite useful facts. Memory capacity, image resolution, and tool-call budgets govern the balance between retained evidence and processing cost.

Results Data

Experiment scores for the textual-memory and optical-memory configurations.

Download Results