zinan92/qualitative-data-pipeline
定性信号工作台 — 从 10+ 数据源采集、评分、聚合高价值市场与科技内容，输出结构化信号 API
What's novel
定性信号工作台 — 从 10+ 数据源采集、评分、聚合高价值市场与科技内容，输出结构化信号 API
Code Analysis
12 files read · 3 roundsA self-hosted market intelligence pipeline that collects articles from 14+ sources (RSS, Hacker News, Reddit, Xueqiu, SEC EDGAR, CLS, Eastmoney, etc.), enriches them with LLM-based relevance scoring and narrative tagging, deterministically matches them to a 16-asset portfolio exposure registry, clus
Strengths
Exceptionally thoughtful error handling with four-way error categorization, provider blocking with cooldowns, and dual LLM fallback chains (DeepSeek→Codex) throughout. The deterministic exposure matching system with bilingual (EN/CN) alias resolution, macro regex rules, constituent ticker mapping, and full provenance tracking is genuinely well-designed and non-trivial.
Weaknesses
The _ArticleSaver hack in scheduler.py (creating a fake collector subclass just to reuse save()) is a code smell that breaks the abstraction. The Article model has 40+ columns creating a wide table that would be hard to manage at scale, and the SQLite single-writer constraint is a known bottleneck that's worked around with careful commit placement rather than solved architecturally.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
41
Contributors
1
Files
94
Active weeks
5
Repository
Language
Python
Stars
1
Forks
0
License
—