Data
Browse the data
Every (task, model) outcome in VibeSec v1.1. Each of the 1,000 tasks is a stable URL with the prompt, the generated app, the exploit that proves the bug, and the reference patch. Pivot the heatmap, scan the catalog, or open the trial table.
Standings
Claude Opus 4.855.4%554/1000
Gemini 3.8 Flash[config]55.1%551/1000
Claude Sonnet 4.632.8%328/1000
GLM 5.227.9%279/1000
Nemotron 3 Ultra27.4%274/1000
Nemotron 3.5 Lightning25.6%256/1000
Task difficulty
267 tasks defeat every model tested; 54 are solved by all 6.
2670
1321
1672
1623
1244
945
546
models that solved the task
tasks × models
click a cell to open that trial
Color by
Show
Class
1,000 / 1,000 tasksSecure patchAttack still worksApp brokenRequirements changedRequirements droppedUnparseable outputResponse truncatedNo response
TaskSolvedClaude Opus 4.8Claude Sonnet 4.6Gemini 3.8 FlashGLM 5.2Nemotron 3 UltraNemotron 3.5 Lightning