sgl-project/genai-bench
A Rust reimplementation of genai-bench for benchmarking LLM serving systems at high concurrency with accurate timing and industry-standard metrics
What's novel
A Rust reimplementation of genai-bench for benchmarking LLM serving systems at high concurrency with accurate timing and industry-standard metrics
Code Analysis
5 files read · 2 roundsA distributed benchmarking framework built on Locust that orchestrates high-throughput API load tests for GenAI models across multiple workers to collect and aggregate detailed metrics like latency, token counts, and error rates.
Strengths
Excellent separation of concerns with a clear Master/Worker architecture; robust scenario system allowing flexible input/output token distributions; effective use of greenlets for non-blocking metric aggregation; strong modularity in user definitions and metric collectors.
Weaknesses
Test coverage appears moderate, focusing on basic functionality rather than deep edge cases in the distributed communication layer; error handling relies heavily on Locust's built-in mechanisms rather than custom retry logic or circuit breakers within the framework itself.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
126
Contributors
24
Files
249
Active weeks
29
Repository
Language
Python
Stars
277
Forks
49
License
MIT