alebels/medianalytics
Platform for the analysis of media and other relevant sources that influence public opinion, utilizing web scraping, NLP and AI.
What's novel
Platform for the analysis of media and other relevant sources that influence public opinion, utilizing web scraping, NLP and AI.
Code Analysis
15 files read · 4 roundsA microservices platform that scrapes news articles from ~25 international media sources daily, uses AI (Google Gemma) to classify each article into 3 sentiments and 3 ideologies from a large predefined taxonomy, performs NLP analysis (word frequency, POS tags, entities) with SpaCy, stores everythin
Strengths
Well-structured microservices architecture with clear separation (scraping, AI, API, frontend), comprehensive per-source scraping configuration, thoughtful SQL query building with CTEs and parameterized queries, and a genuinely large and well-organized ideology/sentiment taxonomy. The scraping pipeline with anti-bot detection, BART summarization for long texts, and the full NLP+AI analysis chain shows real implementation depth.
Weaknesses
The AI classification is a thin wrapper around a single LLM call with a fixed prompt asking for exactly 3 labels from a list — no confidence scores, no multi-pass validation, and the JSON extraction regex is fragile. Test coverage is minimal (only Angular spec files exist, no Python tests), and the SQL query building in filters.py, while functional, mixes query construction logic with service logic in a way that's hard to maintain as filters grow.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
25
Contributors
1
Files
210
Active weeks
16
Repository
Language
Python
Stars
3
Forks
0
License
GPL-3.0