Crawler

Also known as: Bot, Spider, Web Crawler

A program search engines use to crawl the web. Bots visit webpages to collect information and add or update a search engine’s index.

FAQ

Google’s crawlers are called googlebots, while those of Microsoft Bing are called bingbots, to name a few.