InfoPO: Information-Driven Policy Optimization for User-Centric Agents

[Submitted on 28 Feb 2026 (v1), last revised 17 Jun 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Real-world user requests to LLM agents are often underspecified. Agents must interact to acquire missing information and make correct downstream decisions. However, current multi-turn GRPO-based methods often rely on trajectory-level reward computation, which leads to credit assignment problems and insufficient advantage signals within rollout groups. A feasible approach is to identify valuable interaction turns at a fine granularity to drive more targeted learning. To address this, we introduce InfoPO (Information-Driven Policy Optimization), which frames multi-turn interaction as a process of active uncertainty reduction and computes an information-gain reward that credits turns whose feedback measurably changes the agent’s subsequent action distribution compared to a masked-feedback counterfactual. It then combines this signal with task outcomes via an adaptive variance-gated fusion to identify information importance while maintaining task-oriented goal direction. Across diverse tasks, including intent clarification, collaborative coding, and tool-augmented decision making, InfoPO consistently outperforms prompting and multi-turn RL baselines. It also demonstrates robustness under user simulator shifts and generalizes effectively to environment-interactive tasks. Overall, InfoPO provides a principled and scalable mechanism for optimizing complex agent-user collaboration. Code is available at this https URL.

Submission history

From: Fanqi Kong [view email]
[v1] Sat, 28 Feb 2026 13:58:14 UTC (1,337 KB)
[v2] Wed, 17 Jun 2026 05:20:41 UTC (18,385 KB)