Top 5 Videos for Web Crawler System Design Interview

1. System Design distributed web crawler to crawl Billions of web pages
(https://www.youtube.com/watch?v=BKZxZwUgL3Y)
by Tech Dummies Narendra L

Comments: 115
Likes: 1.3K
With 83.4K subscribers on the channels, it’s evident that Narendra L has quite some following among software engineers. The 46-minutes video provides an extensive explanation on designing a web crawler from the perspective of interview questions at Google, Facebook, and Amazon. Narendra has put effort and time to go in depth and create one of the most detailed and comprehensible contents on web crawlers, especially for the high-level components.

2. Design a Web crawler for indexing pages on the internet (SDE — 2 Interview) (https://www.youtube.com/watch?v=shp63TTI1ok) by The Stupid CS Guy

Comments: 7
Likes: 51
Though the video doesn’t have as much following as the first one on the list, it does a good job at preparing the viewers for a system design interview. It’s a mock interview session in which the interviewee is expected to design a basic web crawler with some additional constraints. You’ll find some key questions you can expect in interviews with detailed explanations.

3. Build your own web crawler (https://www.youtube.com/watch?v=rPjt2ilVMzs) by Ahmed Elfakharany

Comments: 7
Likes: 50

Though not created from the perspective of interview questions, the video does give a detailed explanation on creating an application that indexes content on the web to feed it into further applications.

4. Python Scrapy Tutorial | Web Scraping and Crawling Using Scrapy | Edureka (https://www.youtube.com/watch?v=5o9lucMaQLc) by edureka!

Comments: 5
Likes: 316

Despite the fact that this video is more implementation-specific, it would be useful in gaining a technical overview on designing web crawlers. The video introduces Scrapy as a general-purpose web crawler, how to use it to build a basic web crawler, and store the extracted information in a file. The detailed tutorial walks the viewers through the processes involved in building and running the bot.

5. Web crawling (https://www.youtube.com/watch?v=B994RSzlCww) by Techno Spark

Comments: 1
Likes: 11

Though the channel has a very little following and the video barely has 170 views, it’s a detailed explanation on the operation and design of a basic web crawler. It goes in depth to explain the web crawler architecture based on the block system diagram. Additional constraints like the robot exclusion protocol are also explained well. Though it might not be a great idea to prepare for a software engineering interview solely from this video, it can be a helpful extra.

This story is sponsored by Educative.io. Check-out their system-design interview prep course.