ashita-ai/arbiter
Arbiter provides simple APIs, complete observability, and provider-agnostic infrastructure for evaluating LLM outputs. Built on PydanticAI with automatic interaction tracking, multiple evaluators, and extensible architecture.
What's novel
Arbiter provides simple APIs, complete observability, and provider-agnostic infrastructure for evaluating LLM outputs. Built on PydanticAI with automatic interaction tracking, multiple evaluators, and extensible architecture.
Code Analysis
8 files read · 3 roundsAn LLM evaluation framework that scores outputs across multiple dimensions (semantic similarity, factuality, groundedness, relevance, pairwise comparison) using PydanticAI agents, with automatic cost tracking via LiteLLM's bundled pricing database and full interaction observability.
Strengths
Excellent architecture with clean template-method pattern for evaluators, genuine cost tracking integrated at every layer (not just a wrapper), and a well-organized codebase with comprehensive type hints, Pydantic validation, and a rich middleware system. The test suite is extensive (30+ test files, many 15-45KB) covering edge cases, error handling, and integration scenarios.
Weaknesses
The core evaluation logic is primarily prompt engineering (crafting system/user prompts for LLM judges) rather than novel algorithms — the 'evaluation' is delegated to the LLM itself. The LLM client's approach of flattening multi-turn conversations into a single prompt string and routing everything through PydanticAI Agent adds unnecessary complexity for simple completions. The pairwise evaluator and middleware system, while present, couldn't be fully verified due to budget constraints.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
188
Contributors
7
Files
132
Active weeks
8
Repository
Language
Python
Stars
8
Forks
7
License
MIT