← 피드로
[Submitted on 12 Sep 2025 (v1), last revised 2 Sep 2026 (this version, v5)]
Abstract:We examine whether human psychometric questionnaires can serve as reliable tools for characterizing and predicting LLM behavior in everyday user interactions. We analyze eight open-source LLMs by comparing their value and personality profiles derived from two different methods: Likert self-reports on established questionnaires (PVQ-40/21 and BFI-44/10) and generation probabilities over value-laden responses to everyday user queries. The two profiles diverge substantially. Within-construct item consistency, often cited as evidence of stable LLM dispositions, disappears in generation probabilities. We find that established questionnaire items contain explicit lexical cues that allow models to recognize the target construct and respond in alignment-consistent, socially desirable ways, whereas realistic user queries contain far less recognizable cues. In addition, demographic persona prompts shift models’ responses to human questionnaires in ways consistent with real human patterns, but no such shifts appear in the generation setting, highlighting that human questionnaires overestimate LLMs’ ability to faithfully reproduce expected psychological traits when role-playing demographic personas. Overall, our study indicates that questionnaire scores alone should not be treated as evidence of LLMs’ response tendencies in realistic user interactions, and supports generation-probability profiling with ecologically valid items as a complementary behavioral measure.
Submission history
From: Woojung Song [view email]
[v1]
Fri, 12 Sep 2025 09:14:42 UTC (519 KB)
[v2]
Wed, 18 Mar 2026 10:00:16 UTC (155 KB)
[v3]
Fri, 3 Apr 2026 11:04:17 UTC (155 KB)
[v4]
Fri, 29 May 2026 15:27:35 UTC (180 KB)
[v5]
Wed, 2 Sep 2026 18:52:43 UTC (177 KB)
추출 본문 · 출처: arxiv.org · https://arxiv.org/abs/2509.10078