botator.com/en-GB/blog/2018/05/crawlers-and-spiders I'm curious to know if I'm interpreting this wrongly, if the crawler is a crawler crawling websites on a defined schedule or if it's simply using the links within the original web page itself. A: A crawler will use links on the page to go deeper into the site. Take for example if you go to http://stackoverflow.com/ and click the "Questions" link, you'll get to a URL like stackoverflow.com/questions. That URL contains links to specific questions, those links are of the form stackoverflow.com/questions/{id}. Take for example http://stackoverflow.com/questions/5798977. This will get you to http://www.google.com/search?q=stackoverflow.com%2Fquestions%2F5798977. This is the search that Google does, taking your search parameters and turning it into the link that Google will actually go to. The URLs in the response you got back, are the results that Google actually gave you. The URLs in your list, and then further links on those URLs, are the links that Google's algorithm found and crawled (and indexed) and are not actual urls that Google would display to the user. A: First of all crawling is usually performed by a software crawler not a human. A human can only perform browsing and link spamming or spidering. A crawler may have some limitations or restrictions on the links which can be followed, and as a result some links may never be reachable. A human, on the other hand, can click on any link. A crawler can crawl websites from various resources. A human, however, cannot browse all resources or crawl sites with no access. Google and other search engines crawl websites either by following links automatically generated by the software crawlers or by sending their bots manually to those websites. With all this said, there is no such thing as a scheduled crawler. A: An automatic crawler is a computer program that scans a specific website by following links in order to get more information about a specific topic, especially a web page. You can find more information at Search Engine Optimisation: Crawler. A: There are two general types of crawlers: spidering and scraping. Spidering involves crawling an entire site. A spider uses a computer to do this (and often a person). The result is that when you visit a site the spider adds your visit to a list of visits, so that it can later report on the visit. Spiders sometimes need to be told the number of links that are on the page that they've just seen. That way, they can decide which ones to crawl next, and how many to start at. These instructions can either be built into the bot, or in a config file. They can also be specified on a command line or as environment variables. Spiders don't need to follow links. If they can find a page with the same information, they don't need to crawl through every link in that page. For example, here's a fairly regular page on StackOverflow: As you can see, there are a lot of links on that page, but it's not really necessary to crawl through them all. The spider should be able to get the same information by following the "Questions" link. And, if that isn't sufficient, the spider could get its data from the "Questions" page. You can see this happening at the bottom of this screenshot: It's just following the links from the left side of the page, without ever needing to visit the right side. You may be able to tell by now that it's actually much better to just follow links on the page (if possible). Why? Well, that's because those links probably already have the information you need, and the whole page might not even be necessary. Not only will the spider likely get to all the information it needs, but it'll also be much faster at finding it. The other type of spider is a scraping spider. These spiders only scrape the specific information you need out of the site. This is usually done for very large sites. While the rest of the site may still need to be crawled, the scraper can usually only crawl a portion of it. A scraper can usually figure out exactly how much information you need on your own. It only needs to know how many pages to start with, how many to follow on each page, and how much data to download from each page. You might be able to tell by now that a scraper is a fairly specialized tool. They can be slow, often only download data from pages in a specific format, and have to be updated every time a page changes (even a simple link update can change the appearance of the page). Since you're asking about the Google index, what you've probably come across is a scraper. Google, and all other search engines, have their own scrapers that crawl the internet. They may also use other scrapers, so the end result may be different. If you're curious how Google crawled the page you linked to, look here. This is from one of my blog posts from a while back. Google can follow a link by just clicking on it. The bot sees the link and follows it. If a link is on the page, Google crawlers will crawl up to five levels deep of the link structure. For example, if you click on the "Questions" link, the bot will visit all pages on the link and follow the links on each page. If you are a logged in user, Google has a database that allows it to display its personalized results. If you aren't logged in, Google tries to find your IP address so that it can build a database of what you're interested in so that it can give you personalized search results. Google can crawl just about anything, but it is most effective on sites that are static (don't change often). It doesn't work at all if the site relies on JavaScript or Ajax. For these reasons, it's best to use other web development techniques to make sure your website is well-organized.