Wednesday, July 31, 2019

System Design distributed web crawler to crawl Billions of web pages | web crawler system design

Here is the link.

Politeness/ crawl rate
DNS query
Distributed crawling
Priority


No comments:

Post a Comment