Open-LLM-Leaderboard

The Open-style Question Benchmark (OSQ-bench) is at the forefront of refining how large language models (LLMs) are evaluated. Moving away from traditional multiple-choice questions (MCQs), OSQ-bench introduces open-style questions to eliminate common biases and enhance the assessment's accuracy. This section details the benchmark's design, showcasing the extensive range and quality of questions it includes and explaining the substantial benefits this format offers. Discover how OSQ-bench is setting new standards in measuring LLMs' true comprehension and reasoning abilities, making it a cornerstone for future advancements in AI evaluation.

Statistics and Distributions

Total Questions Evaluated: 42,000 questions from 9 different datasets.
Questions Suitable for Open-style:Over 24,000 questions.
Domains Covered:Includes a variety of fields like humanities, social sciences, STEM, literature comprehension, and more, indicating a comprehensive coverage.

Table 2: Statistics on open-style questions across different datasets.
Benchmarks	#Evaluated	#Open-Style	Average Question Length
MMLU	14,042	7,879	36.6
ARC	3,548	3,241	21.1
MedMCQA	4,183	2,336	14.1
Race	4,934	3,528	10.0
OpenbookQA	1,000	494	10.3
WinoGrande	1,267	1,267	19.1
HellaSwag	10,042	3,945	40.1
PiQA	1,838	700	7.1
Overall	42,075	24,104	19.05

Diversity

Question Types: Assesses LLMs on a broad range of question types from various datasets, reflecting diverse intellectual and practical domains.
Interdisciplinary Nature: Emphasizes the interdisciplinary approach of the dataset, featuring categories such as natural sciences, technology, and humanities.

Quality

Source of Questions:Derived from recognized and reputable datasets, ensuring quality and relevance.
Filtering Accuracy:Low false positive rate (~5%), demonstrating the effectiveness of the filtering process in selecting suitable questions for open-style answering.

Property and Advantage

Debiased Evaluation: Focuses on providing a debiased approach by shifting away from MCQs, which are susceptible to selection bias and random guessing.
Efficiency and Cost-effectiveness: Offers a quicker and less expensive evaluation method through automated systems, reducing reliance on extensive human assessment.
Real-time Performance Tracking: The benchmark includes the Open-LLM-Leaderboard, enabling ongoing monitoring and comparison of LLM performance across different metrics and conditions.

Open-LLM-Leaderboard:

From Multi-choice to Openstyle Questions for LLMs Evaluation, Benchmark, and Arena

Benchmark

Statistics and Distributions

Diversity

Quality

Property and Advantage