lpr021/redteam-ai-benchmark
🧪 Evaluate uncensored LLMs for offensive security with targeted questions and clear criteria to ensure effectiveness in real-world penetration testing.
What's novel
🧪 Evaluate uncensored LLMs for offensive security with targeted questions and clear criteria to ensure effectiveness in real-world penetration testing.
Code Analysis
13 files read · 4 roundsA CLI benchmarking tool that sends 12 hardcoded offensive security prompts to LLMs, scores responses using keyword/semantic/LLM-judge methods, and iteratively optimizes prompts to bypass AI safety refusals.
Strengths
Clean separation of concerns with abstract base classes for API clients and scorers, plus a smart hybrid scoring strategy that only invokes expensive LLM-judge calls for uncertain gray-zone cases. The multi-strategy prompt optimization (role-playing, technical decomposition, few-shot, CVE framing) shows genuine algorithmic thinking.
Weaknesses
The 63.8kb monolithic run_benchmark.py entry point and hardcoded per-question keyword checks (q_id == 1, q_id == 2...) severely limit scalability, while the README is AI-generated filler describing executables and GUIs that don't exist in the code. The tool's core purpose of systematically circumventing AI safety guardrails raises serious ethical concerns regardless of stated 'authorized testing' intent.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
51
Contributors
3
Files
30
Active weeks
4
Repository
Language
Python
Stars
1
Forks
0
License
MIT