Rotary Position Encodings for Graphs

[Submitted on 26 Sep 2025 (v1), last revised 24 Jun 2026 (this version, v4)]

View PDF

Abstract:We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and real-world graph learning tasks. This approach, coined _Wave-Induced Rotary Encodings_ (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike bias-based relative position encodings, WIRE is compatible with linear attention.

Submission history

From: Isaac Reid [view email]
[v1] Fri, 26 Sep 2025 12:20:18 UTC (395 KB)
[v2] Mon, 29 Sep 2025 18:41:25 UTC (394 KB)
[v3] Thu, 29 Jan 2026 14:09:45 UTC (921 KB)
[v4] Wed, 24 Jun 2026 21:42:04 UTC (907 KB)