REPOSITORY-LEVEL GPU SOFTWARE ENGINEERING

A cheerful alpaca engineer holding a GPU graphics card and a wrench

SWE-GPU-Bench

Real repositoriesCross-language patchesGPU execution

Can Language Models Solve Real-World GPU Software Engineering Tasks?

Beyond isolated kernels. SWE-GPU-Bench evaluates language models on bug fixes, features, and performance optimizations inside real GPU repositories.

Feng ChenBinbin LiuWenhan HanYin Zheng*

Independent Researcher · * Corresponding author

417Tasks in the paper
30GPU repositories
378General-set tasks
39Perf-Replay tasks

THE CHALLENGE

A kernel is only part of the codebase.

Real GPU software spans CUDA kernels, C++ host code, Python bindings, and build systems. A successful patch must fit all of them—and pass the repository’s tests.

Device codeCUDA · GPU kernels
Host & bindingsC++ · Python
Build & verificationCMake · Bazel · Tests

Real maintenance work

Bug fixes, feature implementations, and performance optimizations drawn from merged pull requests.

Cross-file reasoning

Over a quarter of tasks edit more than four files; over half edit files in two or more languages.

Execution-grounded evaluation

A model receives the base repository and a problem statement. Hidden PR tests grade its candidate patch.

Evaluation workflow · The agent edits the base repository; hidden tests assess correctness. Perf-Replay measures performance only after those tests pass.

Click a figure to enlarge. Press Escape to close.

THE BENCHMARK

From merged pull requests to executable tasks.

The paper describes a five-stage construction pipeline: repository and PR filtering, command synthesis, clean fail-to-pass replay, and manual review before packaging.

Data construction · Rule and LLM filtering → command synthesis → clean replay → task packaging. See the paper for the full verification protocol.
Tasks by source repository
Languages edited per reference patch
Task categories

EXPERIMENTAL RESULTS

From an applicable patch to a passing test.

Patch, Compile, and Test report percentages of attempted tasks. Test is the task resolution rate. Each configuration uses a single pass@1 run, on a task-matched NVIDIA A100 or H100 GPU.

Current general-set evaluation

Paper-reported · Table 2

These five models were evaluated on the updated general set (378 tasks). Agent configurations use Codex CLI or OpenCode; Oracle is single-pass generation with the target files provided.

Scroll horizontally to see all metrics.

Current general-set evaluation
ModelFrameworkPatch %Compile %Test % ↓
GPT-6-AstraCodex CLI100.0100.058.3
GPT-6.1-SolCodex CLI100.0100.058.0
GPT-5.6-SolCodex CLI98.098.056.7
DeepSeek-V4.1-FlashOpenCode100.0100.055.3
DeepSeek-V4-ProOpenCode94.094.043.3

5 models shown

Show historical results

Historical results · retained subsets

Earlier results use retained valid tasks from prior runs. Their denominators differ and are not specified per row in the paper. They are shown separately and must not be read as an identical-task ranking against the current set.

Historical results · retained subsets
ModelFrameworkPatch %Compile %Test % ↓
GPT-5.5Codex CLI100.095.248.9
GPT-5-CodexCodex CLI85.047.033.0
GPT-5.1-CodexCodex CLI74.757.833.0
GPT-5 miniCodex CLI87.856.730.0
DeepSeek-V3.1OpenCode85.158.529.0
Kimi K2.5OpenCode85.053.027.0
DeepSeek-V3.2OpenCode86.756.126.5
DeepSeek-V4-FlashOpenCode88.288.226.3

PERF-REPLAY

Correctness comes before speed.

On 39 performance tasks, a patch must first pass functional tests. Only then is its target performance metric compared with the base commit.

39 performance tasks

A CORRECTNESS-GATED TRACK

Paper-reported Perf-Replay results (Table 3)
ModelPatchCompileTestPerf. gain
Gold patch Reference patch39/3939/3939/396.4%
GPT-5.1-Codex (Codex)39/3915/393/393.3%
GPT-5-Codex (Codex)39/3910/392/392.4%
Kimi K2.5 (OpenCode)35/398/392/393.3%
DeepSeek-V3.2 (OpenCode)39/3912/391/394.3%

Performance gain is the mean relative improvement on each model’s own passing tasks. These small, different subsets do not support a direct cross-model speedup ranking. The gold patch is the reference implementation.

EXPLORE & CITE

Take a closer look.

Paper

The complete manuscript, including experimental details, analysis, and appendices.

Download PDF

Dataset

Task instances and release information on Hugging Face. Check the release version before comparing results.

Open Hugging Face

Evaluation code

The project source repository is currently private.

Not yet public

Citation

@misc{chen2026swegpubench,
  title = {SWE-GPU-Bench: Can Language Models Solve Real-World GPU Software Engineering Tasks?},
  author = {Chen, Feng and Liu, Binbin and Han, Wenhan and Zheng, Yin},
  year = {2026},
  url = {https://chenfengshijie.github.io/SWE-GPU-Bench-website/}
}

Questions about the benchmark? yzheng3xg@gmail.com

Content snapshot · 6 November 2026