SolomonB14D3/confidence-cartography-toolkit
Teacher-forced confidence analysis for language models. pip install confidence-cartography
What's novel
Teacher-forced confidence analysis for language models. pip install confidence-cartography
Code Analysis
11 files read · 3 roundsA research toolkit that measures language model confidence at the token level using teacher-forced analysis to detect false beliefs and hallucinations in generated text.
Strengths
Excellent separation of concerns with clean core library vs application layer. Implements a novel teacher-forced confidence scoring approach with multiple backends (HuggingFace, Ollama). Comprehensive benchmarking infrastructure for evaluating model honesty against human false-belief data.
Weaknesses
Limited test coverage - only basic unit tests exist without integration or stress testing. Some edge cases in chunking logic could be more robust. The Ollama backend relies on approximate methods that may not fully capture the intended confidence metrics.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
5
Contributors
0
Files
48
Active weeks
2
Repository
Language
Python
Stars
1
Forks
0
License
MIT