Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

[Submitted on 30 Sep 2025 (v1), last revised 28 Jul 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning. Within the context of LoRA, a key open question is how to obtain expressive low-rank adapters from over-parameterized spaces. We propose \textit{PrunedLoRA}, a new framework that leverages structured pruning to obtain highly representative low-rank adapters from an over-parameterized initialization. Unlike prior approaches that impose a fixed low-rank budget, PrunedLoRA dynamically prunes less important components during fine-tuning and prevents their reactivation, enabling flexible and adaptive rank allocation. For structured pruning, by minimizing the pruning error for overall loss, we provide fine-grained pruning and recovery updates in a gradient-based pruning strategy with grounded interpretation. We provide the first theoretical analysis of the robustness of structured pruning and provably show that under the impact of weight perturbation, gradient-based pruning is more robust than activation-based pruning with respect to overall loss. Empirically, PrunedLoRA consistently outperforms LoRA and its variants across supervised fine-tuning tasks in mathematical reasoning, code generation, and natural language understanding, and it also demonstrates advantages over existing structured pruning methods across diverse sparsity levels.

Submission history

From: Xin Yu [view email]
[v1] Tue, 30 Sep 2025 19:10:35 UTC (1,393 KB)
[v2] Sat, 1 Nov 2025 04:19:13 UTC (1,626 KB)
[v3] Tue, 28 Jul 2026 23:06:32 UTC (1,486 KB)