zee839/APTBench
🚀 Evaluate base LLMs' agent capabilities in software engineering and deep research with APTBench for efficient, predictive performance insights.
What's novel
🚀 Evaluate base LLMs' agent capabilities in software engineering and deep research with APTBench for efficient, predictive performance insights.
Code Analysis
12 files read · 3 roundsA research benchmark harness that evaluates base (pre-trained) LLMs on agentic tasks—environment setup, deep research planning/citation, and issue fixing—by feeding few-shot prompts and scoring exact-match or ROUGE accuracy.
Strengths
The benchmark covers a genuinely interesting research question (can base models do agentic work?) with a well-structured task configuration system and multi-backend inference support (HF/vLLM/SGLang). The curated dataset with bilingual (EN/ZH) tasks and diverse agentic scenarios provides real research value.
Weaknesses
The README is entirely fabricated, describing a non-existent desktop GUI application with installers and data visualization. The code has significant duplication (three near-identical get_results.py files), no tests, a broken reference to a non-existent script (pred_sglang_api.py), and fragile answer extraction logic (regex/string splitting).
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
17
Contributors
2
Files
64
Active weeks
5
Repository
Language
Python
Stars
1
Forks
0
License
Apache-2.0