Human Grounded Evaluation of Large Language Models for Optical Network Automation

[Submitted on 20 Jul 2026 (v1), last revised 21 Jul 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Large language models (LLMs) are increasingly adopted for network automation, yet their output quality and inference cost can vary substantially across LLM families. We present HuGLEN, a stepwise evaluation pipeline that uses an LLM-as-a-judge together with a small set of expert ratings to enable scalable and reproducible comparison of candidate LLMs, and to rank them using a quality efficiency score (QES). We demonstrate HuGLEN for translating outputs from an explainable artificial intelligence (XAI) model for the optical network quality of transmission (QoT) estimation task into operator-friendly explanations. Our results show that a medium-sized LLM (12B parameters) achieves the highest QES, indicating the best trade-off between explanation quality and efficiency. Overall, HuGLEN reduces the human-labeling burden while supporting consistent model selection for operator-facing automation tasks.

Submission history

From: Kiarash Rezaei [view email]
[v1] Mon, 20 Jul 2026 15:36:00 UTC (400 KB)
[v2] Tue, 21 Jul 2026 09:11:26 UTC (400 KB)