Real maintenance work
Bug fixes, feature implementations, and performance optimizations drawn from merged pull requests.
REPOSITORY-LEVEL GPU SOFTWARE ENGINEERING

SWE-GPU-Bench
Beyond isolated kernels. SWE-GPU-Bench evaluates language models on bug fixes, features, and performance optimizations inside real GPU repositories.
Independent Researcher · * Corresponding author
THE CHALLENGE
Real GPU software spans CUDA kernels, C++ host code, Python bindings, and build systems. A successful patch must fit all of them—and pass the repository’s tests.
Bug fixes, feature implementations, and performance optimizations drawn from merged pull requests.
Over a quarter of tasks edit more than four files; over half edit files in two or more languages.
A model receives the base repository and a problem statement. Hidden PR tests grade its candidate patch.
Click a figure to enlarge. Press Escape to close.
THE BENCHMARK
The paper describes a five-stage construction pipeline: repository and PR filtering, command synthesis, clean fail-to-pass replay, and manual review before packaging.
EXPERIMENTAL RESULTS
Patch, Compile, and Test report percentages of attempted tasks. Test is the task resolution rate. Each configuration uses a single pass@1 run, on a task-matched NVIDIA A100 or H100 GPU.
These five models were evaluated on the updated general set (378 tasks). Agent configurations use Codex CLI or OpenCode; Oracle is single-pass generation with the target files provided.
Scroll horizontally to see all metrics.
| Model | Framework | Patch % | Compile % | Test % ↓ |
|---|---|---|---|---|
| GPT-6-Astra | Codex CLI | 100.0 | 100.0 | 58.3 |
| GPT-6.1-Sol | Codex CLI | 100.0 | 100.0 | 58.0 |
| GPT-5.6-Sol | Codex CLI | 98.0 | 98.0 | 56.7 |
| DeepSeek-V4.1-Flash | OpenCode | 100.0 | 100.0 | 55.3 |
| DeepSeek-V4-Pro | OpenCode | 94.0 | 94.0 | 43.3 |
No models match this search.
5 models shown
Earlier results use retained valid tasks from prior runs. Their denominators differ and are not specified per row in the paper. They are shown separately and must not be read as an identical-task ranking against the current set.
| Model | Framework | Patch % | Compile % | Test % ↓ |
|---|---|---|---|---|
| GPT-5.5 | Codex CLI | 100.0 | 95.2 | 48.9 |
| GPT-5-Codex | Codex CLI | 85.0 | 47.0 | 33.0 |
| GPT-5.1-Codex | Codex CLI | 74.7 | 57.8 | 33.0 |
| GPT-5 mini | Codex CLI | 87.8 | 56.7 | 30.0 |
| DeepSeek-V3.1 | OpenCode | 85.1 | 58.5 | 29.0 |
| Kimi K2.5 | OpenCode | 85.0 | 53.0 | 27.0 |
| DeepSeek-V3.2 | OpenCode | 86.7 | 56.1 | 26.5 |
| DeepSeek-V4-Flash | OpenCode | 88.2 | 88.2 | 26.3 |
No models match this search.
PERF-REPLAY
On 39 performance tasks, a patch must first pass functional tests. Only then is its target performance metric compared with the base commit.
A CORRECTNESS-GATED TRACK
| Model | Patch | Compile | Test | Perf. gain |
|---|---|---|---|---|
| Gold patch Reference patch | 39/39 | 39/39 | 39/39 | 6.4% |
| GPT-5.1-Codex (Codex) | 39/39 | 15/39 | 3/39 | 3.3% |
| GPT-5-Codex (Codex) | 39/39 | 10/39 | 2/39 | 2.4% |
| Kimi K2.5 (OpenCode) | 35/39 | 8/39 | 2/39 | 3.3% |
| DeepSeek-V3.2 (OpenCode) | 39/39 | 12/39 | 1/39 | 4.3% |
Performance gain is the mean relative improvement on each model’s own passing tasks. These small, different subsets do not support a direct cross-model speedup ranking. The gold patch is the reference implementation.
EXPLORE & CITE
The complete manuscript, including experimental details, analysis, and appendices.
Download PDFTask instances and release information on Hugging Face. Check the release version before comparing results.
Open Hugging FaceThe project source repository is currently private.
Not yet public@misc{chen2026swegpubench,
title = {SWE-GPU-Bench: Can Language Models Solve Real-World GPU Software Engineering Tasks?},
author = {Chen, Feng and Liu, Binbin and Han, Wenhan and Zheng, Yin},
year = {2026},
url = {https://chenfengshijie.github.io/SWE-GPU-Bench-website/}
}Questions about the benchmark? yzheng3xg@gmail.com
Content snapshot · 6 November 2026