Distributed crawling
Scrapy workers claim frontier batches from PostgreSQL with FOR UPDATE SKIP LOCKED. A reaper reclaims expired leases, so no URL is ever stranded by a dead worker.
crawl planeNewsCrawl is a distributed crawling and intelligence platform for multilingual newsrooms. It fetches Bangla and English coverage on a lease-aware frontier, archives the raw HTML, strips the noise, kills duplicates in four stages, extracts structure with an LLM, and serves it all through semantic search.
বাংলা ও ইংরেজি — দুই ভাষার সংবাদ একটিই অনুসন্ধান স্তরে।
Five independent planes — control, crawl, processing, query and observability — keep the system resilient under real newsroom load. Nothing reaches into another plane's internals; everything crosses through Postgres, Redis Streams or object storage.
Scrapy workers claim frontier batches from PostgreSQL with FOR UPDATE SKIP LOCKED. A reaper reclaims expired leases, so no URL is ever stranded by a dead worker.
crawl planePlain HTTP first, always. Playwright renders only when a source is flagged requires_browser or the body probe comes back empty — 10–50× cheaper than browsering everything.
cost controlRedis Streams consumer groups move articles through cleaning, language detection, dedup, extraction and embedding. At-least-once delivery, idempotent consumers, dead-letter stream.
processing planeURL hash → content SHA-256 → 64-bit SimHash banding → BGE-M3 cosine. Expensive semantic comparison only ever runs on candidates the cheap stages could not resolve.
qualityGemini by default; OpenAI, Anthropic, DeepSeek and Ollama share one interface. Strict JSON validated by Pydantic, with a heuristic fallback before anything is dead-lettered.
extractionBGE-M3 embeddings in pgvector put Bangla and English in one 1024-dimension space — a Bangla query surfaces the English wire story covering the same event.
query planeA new outlet is a sources row: base URL, discovery feeds, crawl policy and CSS selectors. One generic spider is parameterized per pass — no code, no deploy.
operationsA scheduled worker purges articles past the configured window and cleans up the matching S3 objects, with every purge cycle recorded for audit.
complianceHover or tap a stage to follow an article from discovery to search. Every hand-off is a durable boundary — Postgres rows, Redis Stream entries or S3 objects — so a crash never costs more than a redelivery.
Section pages, sitemaps and RSS feeds are expanded into normalized crawl_urls rows. The frontier lives in PostgreSQL — not Redis — so it survives restarts and stays the single source of truth.
Wire copy gets republished across half a dozen outlets within minutes. Comparing every pair semantically would be ruinous, so each stage only sees what the previous one could not resolve — and the expensive vector comparison runs on a handful of candidates instead of the corpus.
Normalized URL collisions — tracking params, trailing slashes and mirrors resolve to one row.
Hash over normalized title, author, published_at and body catches byte-identical republication.
64-bit fingerprints bucketed into bands surface near-duplicates without an O(n²) sweep.
BGE-M3 vectors settle the remaining candidates — reworded rewrites of the same event.
Bangla is not an afterthought here. URL normalization, content hashing, language detection and indexing were built for bn and en together, so a Bangla query can surface the English wire story covering the same event — and vice versa.
ঘূর্ণিঝড়ে উপকূলীয় জেলায় ত্রাণ বিতরণ
উপকূলে ত্রাণ তৎপরতা জোরদার, ক্ষয়ক্ষতির হিসাব চলছে
Relief distribution widens in cyclone-hit coastal districts
A wedged Playwright worker cannot stall the dashboard. A slow LLM provider cannot block crawling. Each plane owns its process, its failure mode and its backpressure — and they only talk through durable boundaries.
Frontier, jobs, articles and vectors. Redis is coordination only, never the source of truth.
At-least-once delivery, enforced through content hashes and upserts rather than dedup-on-read.
Illegal job or frontier transitions raise instead of silently writing a bad status.
Prometheus metrics, Grafana boards and structured logs across every plane.
Create a sources row through the admin API or the dashboard's Sources page — base URL, allowed domains, discovery feeds, crawl frequency and the CSS selector set for title, body, author and dates. One generic spider is parameterized by that snapshot on every pass, so onboarding needs no code change and no deploy.
The frontier is state you cannot afford to lose. Workers claim batches with FOR UPDATE SKIP LOCKED and hold a lease; a reaper reclaims leases that expire. Redis handles coordination only — politeness tokens, heartbeats, delayed-retry sets and control signals.
Bangla and English articles share one BGE-M3 embedding space, so meaning-based retrieval crosses the script boundary. Query language is auto-detected in the UI, and Bangla queries can be scoped to Bangla articles when you want same-language results only.
Responses are parsed against strict Pydantic schemas. A malformed response falls back to heuristic extraction; only if that also fails does the message go to the processing:dead-letter stream, where it stays inspectable rather than silently lost.
Yes. LLM providers sit behind one interface — Gemini, OpenAI, Anthropic, DeepSeek and Ollama are implemented against it. Embeddings are versioned rather than overwritten: each row records model, dimension and version, so changing models re-embeds incrementally and stays reversible.
Search the live corpus without an account, or sign in to run crawls, inspect queues and audit source health from the control plane.