Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Googlebot not respecting custom crawl rate


rowan194 - 3:51 am on Jun 14, 2011 (gmt 0)


My OP was based on Google's IP range but I've realised this may also include auxiliary bots like Adsense, translate, web accelerator etc. I've checked an alternate log file and based on the user-agent there were 55,460 fetches from Googlebot itself over a 24 hour period.

These are all indeed text/html fetches.


Thread source:: http://www.webmasterworld.com/robots_txt/4325733.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com