rgb-99/nlp-dataset-engine
High-performance streaming engine for processing terabyte-scale NLP datasets: lazy loading, auto-validation, language filtering, sharding, compression, and benchmarking.
What's novel
High-performance streaming engine for processing terabyte-scale NLP datasets: lazy loading, auto-validation, language filtering, sharding, compression, and benchmarking.
Code Analysis
17 files read · 4 roundsA CLI tool that streams CSV/TXT files line-by-line, validates text quality (length, symbol ratio, optional English detection), and writes filtered records to sharded JSONL files with optional GZIP compression and SHA-256 manifest generation.
Strengths
Clean separation of concerns across small focused modules (streamer, validator, sharder, crawler, stats), and the generator-based pipeline genuinely achieves O(1) memory for the read path. The checkpoint/resume and manifest integrity features add practical value for production batch processing.
Weaknesses
The validator has a real bug (max_symbol_ratio parameter ignored, hardcoded 0.30), two modules (loader.py, jsonl_writer.py) are dead code not used by the CLI, and tests cover only 5 of 12 source modules with no integration tests for sharding, checkpointing, or the CLI itself. The 'terabyte-scale' and 'high-performance' claims in the README are marketing — the implementation is a straightforward line-by-line CSV reader with basic string checks.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
32
Contributors
1
Files
38
Active weeks
3
Repository
Language
Python
Stars
1
Forks
0
License
MIT