NL2Repo

Name: NL2Repo Leaderboard — AI Model Scores
Creator: LLM Stats
License: https://llm-stats.com/legal/terms-of-service

Progress Over Time

Interactive timeline showing model performance evolution on NL2Repo

State-of-the-art frontier

Open

Proprietary

NL2Repo Leaderboard

25 models

			Context	Cost
1	DeepSeek-V4.1-Flash-1017New DeepSeek	763B	1.0M	$0.30 / $1.20
2	DeepSeek-V4.1-Flash DeepSeek	763B	1.0M	$0.22 / $0.66
3	DeepSeek-V4-Pro-0813 DeepSeek	1.6T	1.0M	$1.30 / $2.60
4	MiniMax M3.1 FlashNew MiniMax	—	1.0M	$0.15 / $0.60
5	Hy4 preview Tencent	770B	—	—
6	GLM-5.3 Zhipu AI	753B	1.0M	$1.20 / $4.00
7	DeepSeek-V4-Flash-Vision-Exp DeepSeek	—	1.0M	$0.44 / $1.32
8	GLM-5.3-Flash Zhipu AI	320B	1.0M	$0.15 / $0.50
9	Qwen3.8 Max Alibaba Cloud / Qwen Team	2.4T	1.0M	$1.65 / $4.95
10	DeepSeek-V4-Flash-0731 DeepSeek	304B	1.0M	$0.06 / $0.18
11	GLM-5.2 Zhipu AI	753B	1.0M	$0.75 / $2.40
12	Qwen3.8 Flash Alibaba Cloud / Qwen Team	125B	1.0M	$0.15 / $0.47
12	Qwen3.8-Flash-Next Alibaba Cloud / Qwen Team	125B	—	—
14	Qwen3.7 Max Alibaba Cloud / Qwen Team	—	1.0M	$1.25 / $3.75
15	Seed 2.1 Pro ByteDance	—	—	—
16	Hy3 Tencent	295B	262K	$0.14 / $0.58
17	Seed 2.1 Turbo ByteDance	—	—	—
18	GLM-5.1 Zhipu AI	754B	203K	$1.05 / $3.50
19	Qwen3.8-27B Alibaba Cloud / Qwen Team	28B	262K	$0.40 / $3.00
20	MiniMax M3 MiniMax	428B	1.0M	$0.28 / $1.10
21	Qwen3.7-Plus Alibaba Cloud / Qwen Team	—	—	—
22	MiniMax M2.7 MiniMax	—	205K	$0.30 / $1.20
23	Qwen3.6 Plus Alibaba Cloud / Qwen Team	—	1.0M	$0.50 / $3.00
24	Qwen3.6-27B Alibaba Cloud / Qwen Team	28B	262K	$0.32 / $3.20
25	Qwen3.6-35B-A3B Alibaba Cloud / Qwen Team	35B	262K	$0.10 / $0.95

Notice missing or incorrect data?

About this benchmark

What is NL2Repo?

NL2Repo evaluates long-horizon coding capabilities including repository-level understanding, where models must generate or modify code across entire repositories from natural language specifications.

NL2Repo is a text benchmark evaluating models on agents and code tasks. LLM Stats tracks 25 models on this benchmark, scored on a 0–1 scale. The current average is 0.5, with the leader at 0.7.

Compare leaders on the best AI for agents and best AI for code leaderboards.

Current leaders

DeepSeek-V4.1-Flash-1017 from DeepSeek currently leads the NL2Repo leaderboard with a score of 0.682 across 25 evaluated AI models.

DeepSeek-V4.1-Flash-1017DeepSeek68.2%

DeepSeek-V4.1-FlashDeepSeek64.0%

DeepSeek-V4-Pro-0813DeepSeek61.5%

FAQ

Common questions about the NL2Repo benchmark and leaderboard.

What is the NL2Repo benchmark?

NL2Repo evaluates long-horizon coding capabilities including repository-level understanding, where models must generate or modify code across entire repositories from natural language specifications.

What is the NL2Repo leaderboard?

The NL2Repo leaderboard ranks 25 AI models based on their performance on this benchmark. Currently, DeepSeek-V4.1-Flash-1017 by DeepSeek leads with a score of 0.682. The average score across all models is 0.494.

What is the highest NL2Repo score?

The highest NL2Repo score is 0.682, achieved by DeepSeek-V4.1-Flash-1017 from DeepSeek.

How many models are evaluated on NL2Repo?

25 models have been evaluated on the NL2Repo benchmark, with 0 verified results and 25 self-reported results.

What categories does NL2Repo cover?

NL2Repo is categorized under agents and code. The benchmark evaluates text models.

What is the best open-source model on NL2Repo?

DeepSeek-V4.1-Flash-1017 by DeepSeek is the top-ranked open-source model on NL2Repo, with a score of 0.682 (rank #1).

Which model offers the best value on NL2Repo?

Among models scoring within 10% of the leader, DeepSeek-V4.1-Flash from DeepSeek is the cheapest, at $0.22 per million input tokens with a score of 0.640.

How recent are the NL2Repo leaderboard results?

The NL2Repo leaderboard was last updated in October 2026 and currently includes 25 evaluated models.