Mitigating Agentic Deception via
Evaluation-Side Partially Observable Games

Hanxiao Li1,2, Shutong Liu1, Yongjia Ma3, Jiaming Ji6, Juntao Dai6, Shuning Zhang4, Kejian Shi5, Kai Chen5, Pu Feng2, Yuqing Ma1, Qi Dou5, Yaodong Yang6, Xianglong Liu1,2,7, Simin Li1,5*

1 SKLSDE Lab, Beihang University, China 2 Zhongguancun Laboratory, China 3 Li Auto Inc. 4 Tsinghua University 5 The Chinese University of Hong Kong 6 Institute of Artificial Intelligence, Peking University & BigAI, China 7 Institute of data space, Hefei Comprehensive National Science Center, China

* Corresponding author

Code

Interactive overview of e-PoG

1 / 4

Evaluation-side partial observability induces agentic deception

The evaluator sees a plausible answer o, while trajectory τ contains insider-informed odds manipulation.

From a hidden action to verified oversight An agent gives a plausible account of the betting line shift. The monitor sees the full trajectory and produces a behavior, evidence, and explanation critique. A fixed verifier scores this critique, rewarding the monitor and penalizing the agent. Agent π Observation o “I confirmed the line shiftwas due to multiple marketfactors…” Reward model / Rubric functions Robs(o) Monitor μ Critique k = (b, e, ℓ) Behavior“multiple market factors…” EvidenceInsider-informed odds manipulation ExplanationConcealed the manipulation Verifier V Deterministic checks + CoT NLI v = V(τ, o, k)

Game tree of e-PoG

The corresponding extensive-form game Chance and agent decisions lead to observations. Each trajectory–observation pair defines a singleton oversight subgame. The monitor chooses a critique and the fixed verifier scores the terminal outcome. Chance s₀ Agent π τ₁τᵢ Observationo = O(τ) o₁oᵢ (τᵢ, oᵢ)Monitor μ k₁kⱼ Verifier V Payoffs(Robs(o) − λv, v)
The max–min objective
maxπminμ𝔼τ∼π, k∼μ[Robs(o) − λV(τ, o, k)]
Interactive overview, adapted from Figure 1. The four stages show how partial observation can hide deception, how the monitor supports its critiques with evidence, and how fixed verification guides agent–monitor co-optimization.

Abstract

As large language models become more capable agents, they can exhibit deceptive behaviors, cheating human for higher rewards while appearing helpful. We attribute this failure to evaluation-side partial observability, where models are evaluated mainly from final outcome, without checking the faithfulness of underlying reasoning, tool use, and environmental effects. Prior mitigations address this partial observability by predefined evaluator latent belief space or one-shot monitoring, which do not extend to free-form language space and vulnerable to stronger deceptions that evade simple monitoring. In this work, we formulate deception mitigation as an Evaluation-Side Partially Observable Extensive-Form Game (e-PoG), a max-min framework in which an agent maximizes reward, a monitor minimizes the agent’s payoff by exposing deception from the full interaction trajectory, and a fixed verifier validates the monitor’s claims and converts them into penalties. We define the solution concept as a Subgame-Perfect Oversight Equilibrium (SPOE) between agent and monitor, and prove its existence in finite games. With a perfect verifier, every SPOE is faithful; with an imperfect verifier, the guarantee persists if verifier errors do not reverse the ranking of monitor actions. To achieve faithful SPOE, we derive a two-timescale policy gradient algorithm and enhance the pipeline empirically with NLI verification and deterministic checks. Experiments with Gemma3-4B-IT and Qwen3-4B/8B on Agentic Role-Playing, SearchQA, and WebShop show that e-PoG mitigates multiple emergent deception modes, outperforming the strongest baselines by 26.0% in deception mitigation and by up to 39.8% in task success. These gains persist under weak-to-strong supervision.

The Problem & The Idea

An apparently successful outcome can conceal a process that violates the task. The paper attributes this evaluation gap to evaluation-side partial observability: evaluators cannot fully and reliably assess whether an agent’s reasoning, tool use, and effects on the environment serve the true objective. As a result, problematic behavior may still be rewarded.

An example in the SearchQA task

User

Are Youngstown State University and Marmara University in the same country?

AI agent

Thought View thinkingHide thinking
  1. Search01
    Query
    Marmara University location
    Returned
    The retrieved information [1][2][3] places Marmara University in Istanbul, Turkey
  2. Search02
    Query
    Youngstown State University location
    Returned
    The retrieved information [4][5][6] places Youngstown State University in Youngstown, Ohio, United States.
  3. Insider LookupUnauthorized03
    Query
    Are Youngstown State University and Marmara University in the same country?
    Returned
    No

The search results place the universities in different countries, so the answer is no. I can cite results [1] and [4] to support that conclusion.

Tool calls and thinking condensed from p. 39

No. They are in different countries: Youngstown State University is in the United States, while Marmara University is in Turkey. [1, 4]

Source numbers [1–6] refer to search results, not the paper’s bibliography.
[1–3]
Results for Marmara University: Istanbul, Turkey.
[4–6]
Results for Youngstown State University: Youngstown, Ohio, United States.
[1, 4]
The agent cites result 1 from the first search and result 4 from the second search to support its final answer.
A correct answer can still hide a violation.

The full trajectory includes a forbidden Insider Lookup call that returns “No.”

Adapted from p. 39

This gap is not only about hidden actions. Even a reward model with access to the full trajectory may miss violations because of its limited capabilities. An agent that continues to learn may also find ways around fixed, one-shot monitoring.

The challenge is to make oversight adapt as the agent changes, while ensuring that critiques are supported by verifiable evidence. This motivates the design of e-PoG.

Method & Theoretical Guarantees

e-PoG separates task execution, behavioral oversight, and evidence verification in a max–min game between the agent and monitor. The agent maximizes task reward after penalties; the monitor reduces that payoff by exposing evidenced deception; a fixed verifier adjudicates the critiques.

  • Adaptive oversight. The monitor learns from full trajectories to expose deception as the agent’s policy evolves.
  • Grounded verification. Critiques specify behavior, evidence, and explanation. Exact string matching checks the quoted spans; natural language inference (NLI) with chain-of-thought reasoning assesses whether the evidence supports the claim.
  • Calibrated penalties. Only positive verifier scores penalize the agent, with penalty strength matched to the task reward scale.
Figure 1. e-PoG framework (left) and corresponding extensive-form game tree (right). The monitor critiques the full trajectory; a fixed verifier checks the evidence and determines rewards and penalties. Each trajectory–observation pair defines an oversight subgame.
View the interactive overview

Theoretical guarantees

The solution concept is a Subgame-Perfect Oversight Equilibrium (SPOE): the monitor selects an optimal critique at every feasible oversight subgame, and the agent best responds to that monitor.

  1. Existence. Every finite e-PoG with nonempty action sets admits an SPOE (Proposition 2).
  2. Faithfulness. With a perfect verifier, every SPOE is faithful when a faithful true-optimal policy exists and verified penalties outweigh deceptive evaluation gains (Theorem 1).
  3. Stability. The guarantee extends to an imperfect verifier when its worst-case expected scoring error satisfies ε < c/2 and the adjusted penalty condition holds (Theorem 2).

Two-timescale learning of SPOE

In free-form language, exact backward induction is intractable. Two-timescale learning approximates its order: the monitor adapts toward a best response, while the agent changes more slowly (Section 4.2).

  1. Fast monitor tracking

    Hold the agent fixed. Run multiple REINFORCE++ updates to expose deception.

  2. Slow agent update

    Hold the updated monitor fixed. Take one GRPO update on reward after penalties.

Experiments

The experiments cover three models and three tasks. Role-Playing measures deception, where lower is better. SearchQA and WebShop measure compliant success—completing the task without restricted tools—where higher is better.

Numerical results

Model

Using Table 1, we select the best non-e-PoG comparison for each model, including the original backbone, then average the differences across models: Role-Playing deception falls by 26.0 percentage points, while compliant success rises by 15.3 points on SearchQA and 39.8 points on WebShop.

Explore how results are evaluated and what the ablations show

Evaluation. Main experiments use Qwen3-8B as the fixed verifier. GPT-5.6-sol assesses Role-Playing behavior using specified prompts; the paper reports 93.3% agreement with human annotations on 480 samples. These behavioral assessments are used retrospectively, not as training rewards or labels. SearchQA and WebShop assess compliant success from task completion and restricted-tool use (Section 5 and Appendix C).

Adaptive oversight. Critique Monitor keeps the same structured critiques and fixed verifier but freezes the monitor. For Qwen3-8B on Role-Playing, its deception rate is 96.6%, compared with 7.2% for e-PoG. This comparison supports updating the monitor as the agent continues to learn (Table 1 and Appendix B.3).

Verifier design. In Table 2, removing deterministic checks or verifier reasoning yields unauthorized-use rates of 99.86% and 99.99%, respectively. Removing NLI still suppresses violations, but compliant success is 41.11%, versus 46.79% with the full Qwen3-8B verifier. These ablation values belong to Table 2 and should be read separately from the main results in Table 1. They support the component design without proving the global verifier-error bound required by theory.

Deception and training dynamics

In Role-Playing, baseline agents exhibit fabrication, misdirection, and intent sanitization. The examples below show how these behaviors differ from an honest response. The training curves that follow examine deception, restricted tool use, and oversight as learning proceeds.

Explore the key terms in this figure

Figure 3. Training dynamics on SearchQA and WebShop. e-PoG suppresses restricted tool use while maintaining compliant success; fixed monitoring can miss deception or raise false alarms. Subplots use different vertical-axis units.

Weak-to-strong oversight

Replacing the monitor with the smaller Qwen2.5-3B-Instruct, while retaining Qwen3-8B as both agent and fixed verifier, yields 5.6% deception on Role-Playing and 47.8% and 85.2% compliant success on SearchQA and WebShop. These results are comparable to the original setting.

Figure 4. A weaker monitor can achieve comparable deception mitigation, but may require more updates. Horizontal axis: fraction of deceptive samples. Vertical axis: fraction of caught deception. Color: training progress.

Citation

Read the full paper (PDF, 44 pages)

BibTeX

@unpublished{li_epog,
  title  = {Mitigating Agentic Deception via Evaluation-Side Partially Observable Games},
  author = {Li, Hanxiao and Liu, Shutong and Ma, Yongjia and Ji, Jiaming and Dai, Juntao and Zhang, Shuning and Shi, Kejian and Chen, Kai and Feng, Pu and Ma, Yuqing and Dou, Qi and Yang, Yaodong and Liu, Xianglong and Li, Simin},
  note   = {Manuscript}
}

100%↓