SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution

[Submitted on 13 Mar 2026 (v1), last revised 21 Jul 2026 (this version, v4)]

Authors:Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang, Dhruv Parikh, Rajgopal Kannan, Le Cong, Mengdi Wang, Qian Zhang, Viktor Prasanna, Xiangru Tang, Xingyao Wang

View PDF HTML (experimental)

Abstract:Real-world software must continuously evolve to meet ever-changing and open-ended requirements. AI agents, increasingly deployed as long-running systems, are now entrusted to drive this evolution. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable SWE-Milestone, a benchmark that evaluates agents on streams of milestone-level tasks, requiring them to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to 38.03% in continuous settings, exposing agents’ profound struggle with long-term maintenance and error propagation.

Submission history

From: Gangda Deng [view email]
[v1] Fri, 13 Mar 2026 03:20:40 UTC (3,430 KB)
[v2] Fri, 5 Jun 2026 11:07:19 UTC (4,271 KB)
[v3] Fri, 10 Jul 2026 10:31:50 UTC (4,298 KB)
[v4] Tue, 21 Jul 2026 11:31:57 UTC (3,608 KB)