ggteixeira/corpus-cleaner
Linguistic tool (made by a linguist, for linguists) that scraps corpora, automatically cleans it up, and generates n-grams.
What's novel
Linguistic tool (made by a linguist, for linguists) that scraps corpora, automatically cleans it up, and generates n-grams.
Code Analysis
12 files read · 3 roundsA collection of small, buggy Python scripts that generate n-gram candidates from a Portuguese software-development corpus, with the actual terminology extraction and hyponym identification done manually and stored as data files.
Strengths
The project contains a well-curated Portuguese corpus of software development texts and a manually validated glossary of domain terms that could be useful as reference data. The n-gram pipeline, while naive, demonstrates a basic understanding of the terminology extraction workflow.
Weaknesses
The code is minimal, buggy (logic errors in the scraper, O(n²) string operations, substring-based stopword filtering), has no tests, no error handling, no package structure, and the 'transitivity' check is a hardcoded no-op. The README describes only the scraper and completely omits the n-gram pipeline, hyponym extraction, and terminology work that constitute 95% of the project.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
10
Contributors
2
Files
150
Active weeks
1
Repository
Language
Python
Stars
1
Forks
0
License
—