rahul-sg/HondaResearchLabs_DSC180A-Eval-Systems-Of-NextGen-LLMs
Domain-aware LLM summary evaluation and iterative refinement pipeline with hybrid risk-adjusted scoring (UCSD DSC180 capstone).
What's novel
Domain-aware LLM summary evaluation and iterative refinement pipeline with hybrid risk-adjusted scoring (UCSD DSC180 capstone).
Code Analysis
14 files read · 4 roundsAn end-to-end pipeline that generates LLM summaries of lecture PDFs, iteratively refines them using lever-based criterion-driven stopping, and evaluates quality through domain-aware rubric judging, deterministic TF-IDF signals, pairwise comparison, and hybrid hallucination-damped scoring.
Strengths
The lever-based refinement controller with progressive signal thresholds, plateau detection, stalled-trend detection, and decision-table stopping is genuinely non-trivial and well-structured. The multi-layered scoring (domain-aware rubric + deterministic signals + hallucination damping with domain-specific alpha multipliers) shows real research depth beyond a simple prompt-and-score loop.
Weaknesses
The hallucination detection is a crude TF-IDF cosine similarity threshold (0.25) that will produce many false positives/negatives, and the domain detection is purely keyword-based with no LLM involvement. There are no proper unit tests — only ad-hoc experiment scripts in dev_notes/ — and the LLM client creates a new OpenAI instance on every call.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
61
Contributors
5
Files
596
Active weeks
8
Repository
Language
Python
Stars
1
Forks
0
License
—