What is the RepoQA leaderboard?

The RepoQA leaderboard ranks 2 AI models based on their performance on this benchmark. Currently, Phi-3.5-MoE-instruct by Microsoft leads with a score of 0.850. The average score across all models is 0.810.

What is the highest RepoQA score?

The highest RepoQA score is 0.850, achieved by Phi-3.5-MoE-instruct from Microsoft.

How many models are evaluated on RepoQA?

2 models have been evaluated on the RepoQA benchmark, with 0 verified results and 2 self-reported results.

Where can I find the RepoQA paper?

The RepoQA paper is available at https://arxiv.org/abs/2406.06025. The paper details the methodology, dataset construction, and evaluation criteria.

What categories does RepoQA cover?

RepoQA is categorized under long context, reasoning, and code. The benchmark evaluates text models with multilingual support.

All benchmarks

RepoQA

RepoQA is a benchmark for evaluating long-context code understanding capabilities of Large Language Models through the Searching Needle Function (SNF) task, where LLMs must locate specific functions in code repositories using natural language descriptions. The benchmark contains 500 code search tasks spanning 50 repositories across 5 modern programming languages (Python, Java, TypeScript, C++, and Rust), tested on 26 general and code-specific LLMs to assess their ability to comprehend and navigate code repositories.

Phi-3.5-MoE-instruct from Microsoft currently leads the RepoQA leaderboard with a score of 0.850 across 2 evaluated AI models.

Paper