How I built the distributed crawling and indexing infrastructure behind Kagi's search engine — Frontera, Kafka, Redis Bloom deduplication, Vespa HA, and the trade-offs of running it in production.