Does Capability Transfer to Subjective Behavior — and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm

[Submitted on 27 May 2026 (v1), last revised 9 Jun 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Benchmarking is mature where answers are verifiable — math, code, reasoning — but the fastest-growing uses of LLMs are subjective and human-facing: companionship, emotional support, counseling. There the default validity test, correlating a metric to human judgment, has no stable anchor: inter-rater agreement is low, structured by annotator identity, barely reproducible, and length-biased. So we cannot answer the question that matters: does capability that scales on objective benchmarks transfer to subjective behavior, and would our instruments even tell us if it did not? We build an instrument for this regime and report what it reveals at the frontier. We contribute, first, a self-evolving instrument that selects and then authors its own behavioral dimensions under a multiplicative anti-gaming fitness, self-halting when it stops improving; second, a trust-by-construction paradigm that earns belief through three certificates established without a human gold standard, where human raters saturate (rho ~ 0.45); and third, the finding it makes visible — capability transfer is dissociable. Across 49 models, 8 families, and 24 months, subjective behaviors are where objective-benchmark scaling fails to carry over: the sharpest case, advice-restraint (knowing when not to give advice), is the frontier’s universal-lowest dimension, and at gpt-4.1->gpt-5 it ran backwards while the aggregate score hid it — a regression one instruction recovers. Warm restraint is moved by model generation, not by raw scale, MoE width, inference budget, or reasoning mode; the open-weight Pareto frontier matches closed flagships at ~10-80x lower per-call cost; and four judge families replicate the rubric on held-out human ESConv conversations. Data, code, the locked rubric, and judge prompts will be released upon publication.

Submission history

From: Rapheal (Yuming) Huang [view email]
[v1] Wed, 27 May 2026 03:41:11 UTC (732 KB)
[v2] Tue, 9 Jun 2026 02:24:01 UTC (1,347 KB)