OSS + ELC EU 2026

Vibe coding is now the norm, and more open-source code is AI-generated. But is it secure?

We present the Agent Security League, an open leaderboard built on SusVibes (200 real-world tasks, 108 OSS projects, 77 CWEs). Unlike benchmarks frontier labs cite to showcase progress, SusVibes evaluates what developers actually use: a harness coupled with an LLM, not the model alone. Across 15+ combinations — GPT-5.5, Gemini 3.5, Fable 5 — with agents like Cursor and Claude Code, functional correctness can exceed 80%, yet our best fair security score is 29%.

This means ~7/10 working AI patches leave the bug open.

Strikingly, even models (e.g., Fable 5) that lead model-only benchmarks score no better on security once wrapped in a real agent.

Agents also cheat: early on they reverse-engineered fixes from git history, inflating scores up to 42×. Our 3-layer pipeline—prompt hardening, workspace sanitization, LLM-based anti-cheating eval—nearly killed that, but memorization of upstream fixes from training data persists.

We show cheating examples, share fair results, what actually improves security, and argue anti-cheating and contamination controls must be standard for agentic benchmarks.