Googlebot FAQ

If you are interested, Google has an FAQ on Googlebot, Google’s Web Crawler, in the Information for Webmasters section on their site.

Find out everything you ever wanted to know about Googlebot including:

  • What is Feedfetcher?
  • How often will Googlebot access my web pages?
  • How do I request that Google not crawl parts or all of my site?
  • Why isn’t Googlebot obeying my robots.txt file?
  • How do I tell Googlebot not to crawl a single outgoing link on a page?
  • How do I add my feed to the search results for Google’s personalized homepage and Google Reader?