Open-LLM-Leaderboard

Discover our innovative approach, which moves beyond traditional Multiple-Choice Questions (MCQs) to Open-Style Questions. This shift aims to eliminate inherent biases and random guessing prevalent in MCQs, providing a clearer insight into the true capabilities of LLMs.

Introduction:

Large language models (LLMs) excel at various natural language processing tasks but need robust evaluation strategies to assess their performance accurately. Traditionally, MCQs have been used for this purpose. However, they are prone to selection bias and random guessing. This paper presents a new approach by transitioning from MCQs to open-style questions, aiming to provide a more accurate assessment of LLM capabilities. We introduce both the Open-LLM-Leaderboard and a new benchmark to evaluate and compare the performance of different LLMs.

Beyond Multiple-Choice Questions:

Multiple-choice questions (MCQs) are frequently used to assess large language models (LLMs). Unfortunately, MCQs can lead to biases due to inherent unbalanced probabilities influencing predictions. Our research introduces a new benchmark through entirely open-style questions, shifting away from MCQs to better reflect true LLM capabilities.

To fundamentally eliminate selection bias and random guessing in LLMs, in this work, we build an open-style question benchmark for LLM evaluation. Leveraging this benchmark, we present the Open-LLM-Leaderboard, a new automated framework designed to refine the assessment process of LLMs.

Key Findings:

Reduction of Selection Bias: Open-style questions eliminate the tendency of LLMs to favor certain answer choices, reducing selection bias.
Minimization of Random Guessing: Open-style questions prevent LLMs from guessing answers, providing a clearer picture of their true knowledge and capabilities.
New Benchmark: The Open-LLM-Benchmark provides a comprehensive evaluation framework using open-style questions across various datasets.
Leaderboard Insights: The Open-LLM-Leaderboard tracks the performance of various LLMs, with GPT-4o currently holding the top position, offering a clear comparison of their capabilities.

Methodology:

Defining Open-style Questions:
Open-style questions require models to generate answers without being constrained by predetermined choices, aiming to assess the model’s ability to generate coherent and contextually appropriate responses. This approach helps avoid selection bias and random guessing inherent in MCQs.

Automatic Open-style Question Filtering and Generation:

Multi-stage Filtering and Postprocessing: We implemented a multi-stage filtering process to convert MCQs to open-style questions. This involves:
- Stage 1: Preliminary Filter using Binary Classification: We initially classify questions as either convertible or non-convertible using a binary classification prompt. Questions that rely heavily on their choices are marked as non-convertible.
- Stage 2: Confidence Score Assignment: Convertible questions are then assigned a confidence score (1-10) indicating the likelihood of being answered in an open-style format. Questions below a certain threshold are excluded, ensuring only the most suitable questions are selected.
Open-style Question Answer Evaluation: We designed customized prompts to evaluate the correctness of LLM responses to open-style questions. The evaluation involve:
- Using the correct MCQ answer as the ground truth.
Validation of Automatic Evaluation Strategy: To validate our approach, we manually checked a random sample of 100 results from the automatic evaluation, confirming an error rate of less than 3%.
Comprehensive Analysis and Ranking: We conducted a thorough assessment of well-recognized large language models (LLMs), including GPT-4o, GPT-4, GPT-3.5, Claude-3 Opus, Gemini-Pro, and Mistral-Large, using our benchmark. The performance of GPT-4o demonstrates its leading edge with an accuracy of 70.15%, indicating its robustness in open-style question answering tasks compared to other models. It is followed by GPT-4-1106-preview with 65.93%, and Claude-3 Opus with 62.68%. These results highlight the advanced capabilities of the GPT-4 series. Mid-tier models like Mistral-Large and GPT-3.5 perform well but are not on par with the top performers. On the other hand, models like Gemini 1.0 Pro and Llama3-70b-Instruct lag behind in their capability to answer open-style questions. The performance evaluation of smaller-scale LLMs reveals that Qwen1.5 leads overall.

@article{myrzakhan2024openllmleaderboard,
  title={Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena},
  author={Aidar Myrzakhan, Sondos Mahmoud Bsharat, Zhiqiang Shen},
  journal={arXiv preprint arXiv:2406.07545},
  year={2024},
}

Open-LLM-Leaderboard:

From Multi-choice to Openstyle Questions for LLMs Evaluation, Benchmark, and Arena

Welcome to Our Research titled
'Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena'

Discover our innovative approach, which moves beyond traditional Multiple-Choice Questions (MCQs) to Open-Style Questions. This shift aims to eliminate inherent biases and random guessing prevalent in MCQs, providing a clearer insight into the true capabilities of LLMs.

Introduction:

Beyond Multiple-Choice Questions:

Key Findings:

Methodology:

Automatic Open-style Question Filtering and Generation:

Citation.

Welcome to Our Research titled 'Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena'

Discover our innovative approach, which moves beyond traditional Multiple-Choice Questions (MCQs) to Open-Style Questions. This shift aims to eliminate inherent biases and random guessing prevalent in MCQs, providing a clearer insight into the true capabilities of LLMs.

Introduction:

Beyond Multiple-Choice Questions:

Key Findings:

Methodology:

Automatic Open-style Question Filtering and Generation:

Citation.

Welcome to Our Research titled
'Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena'