DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain

[Submitted on 18 Apr 2025 (v1), last revised 26 Jun 2026 (this version, v4)]

Authors:Enhao Huang, Pengyu Sun, Shuxun Wang, Zixin Lin, Alex Chen, Kaichun Hu, Joey Ouyang, Frank Li, Zhiyu Zhang, Haobo Wang, Yiming Li, Zhan Qin, James Yi, Gang Zhao, Ziang Ling, Lowes Yang

View PDF HTML (experimental)

Abstract:The Web3 ecosystem, underpinned by cryptographic primitives and decentralized consensus, represents a high-stakes environment where software vulnerabilities and incentive misalignments translate directly into financial loss. As Large Language Models (LLMs) are increasingly integrated into this domain for tasks ranging from smart contract auditing to decentralized finance analytics, ensuring their reliability is paramount. However, general-purpose benchmarks fail to capture the specialized reasoning required for these adversarial and protocol-driven settings. To bridge this gap, we introduce DMind Benchmark, a comprehensive evaluation suite designed to rigorously assess LLM proficiency across the Web3 stack. DMind Benchmark encompasses nine distinct subdomains (spanning infrastructure, smart contracts, token economics, etc.) and combines objective knowledge retrieval with complex open-ended reasoning tasks that emulate real-world operational challenges. We conduct an extensive evaluation of 31 leading proprietary and open-weights models, employing a contamination-aware pipeline and verifying the statistical robustness of our scoring protocol through rigorous cross-judge consistency checks. Our analysis reveals a critical dichotomy: while models demonstrate competence in foundational infrastructure concepts, they exhibit significant vulnerabilities in high-reasoning tasks such as security auditing. Furthermore, we provide a Pareto analysis to guide cost-effective deployment and demonstrate through adversarial experiments that high performance on DMind Benchmark necessitates genuine reasoning rather than superficial memorization. Since its open-source release in April 2025, DMind Benchmark achieved the #1 trending position on Hugging Face for nearly a week and accumulated over 13k downloads by June 2026, establishing itself as a standard for advancing secure and trustworthy AI in Web3.

Submission history

From: Yihui Yang [view email]
[v1] Fri, 18 Apr 2025 16:40:39 UTC (1,541 KB)
[v2] Fri, 16 May 2025 12:00:59 UTC (1,085 KB)
[v3] Tue, 4 Nov 2025 16:26:20 UTC (5,861 KB)
[v4] Fri, 26 Jun 2026 10:59:58 UTC (17,655 KB)