plater7/docrawl
Web crawler para sitios de documentación — convierte páginas a Markdown limpio usando Playwright, markdownify y Ollama (local) u OpenRouter/OpenCode (cloud)
What's novel
Web crawler para sitios de documentación — convierte páginas a Markdown limpio usando Playwright, markdownify y Ollama (local) u OpenRouter/OpenCode (cloud)
Code Analysis
10 files read · 3 roundsA FastAPI-based web documentation crawler that transforms documentation websites into clean Markdown or structured JSON through a multi-stage pipeline of URL discovery, LLM-based filtering, multi-fallback page scraping, DOM noise removal, and optional LLM cleanup.
Strengths
Genuinely thoughtful engineering in the scraping fallback chain (native markdown → proxy → HTTP fast-path → Playwright), adaptive LLM cleanup classification with heuristics for broken tables/LaTeX/code density, and robust job management with pause/resume, SSE streaming, and content deduplication. Security is well-handled with SSRF validation, defusedxml, API key auth, and rate limiting.
Weaknesses
The LLM cleanup is fundamentally prompt-dependent with no deterministic post-processing to verify content integrity, and the structured JSON output is a basic BeautifulSoup flattening that loses nested structure. The reasoning_model is declared but unused, and LLM calls are sequential per-page with no batching.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
281
Contributors
4
Files
153
Active weeks
5
Repository
Language
Python
Stars
1
Forks
0
License
MIT