← 피드로
[Submitted on 7 Aug 2026 (v1), last revised 25 Sep 2026 (this version, v4)]
Abstract:Replay-free parallelized Q-learning removes the large experience replay buffers and target networks used by conventional deep Q-learning, but the role of network architecture in this training regime remains comparatively underexplored. We investigate this question through a progressive three-phase study within the Parallelized Q-Network (PQN) framework. First, we compare eight convolutional encoder topologies on Atari-57 under a common training protocol while jointly considering performance and computational complexity. Second, we integrate Hadamax-style multiplicative feature interactions and explicit pooling into the selected encoder hierarchy. Third, with the visual representation fixed, we compare complete categorical-dueling, ensemble-dueling, and categorical ensemble-dueling value-estimation configurations. The resulting architecture, Aftab, achieves an interquartile mean human-normalized score of $6.592$ on Atari-57, compared with $2.715$ for our independently rerun PQN reference, with a game-level Probability of Improvement of $0.86$. After completing all architecture selection on Atari-57, we evaluate Aftab on Procgen Hard. Aftab achieves a terminal IQM normalized score of $0.418$ compared with $0.382$ for PQN and increases the normalized area under the learning curve from $0.216$ to $0.541$, although terminal performance remains heterogeneous across environments. These results show that visual topology, multiplicative representation, and downstream value-estimation design can substantially affect replay-free Q-learning, and that their benefits should be evaluated jointly with computational complexity. The complete Aftab framework, including model definitions, training configurations, reproducibility settings, and raw experimental logs, is open-sourced at this https URL
Submission history
From: Taha Shieenavaz [view email]
[v1]
Fri, 7 Aug 2026 15:29:15 UTC (8,445 KB)
[v2]
Thu, 13 Aug 2026 10:45:39 UTC (8,830 KB)
[v3]
Thu, 24 Sep 2026 13:18:12 UTC (7,616 KB)
[v4]
Fri, 25 Sep 2026 10:35:20 UTC (7,616 KB)
추출 본문 · 출처: arxiv.org · https://arxiv.org/abs/2608.07335