Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Googlebot not respecting custom crawl rate


rowan194 - 3:38 am on Jun 22, 2011 (gmt 0)


Here's what I ended up doing.

1. Set preferred domain to www.example.com in GWT

2. Configured web server to 301 redirect all requests for example.com to www.example.com

3. Disabled custom crawl rate in GWT for example.com. (I don't mind if GBot sends 100k requests per day that 301, because this doesn't involve any database I/O, and the redirected URLs get queued rather than fetched immediately)

4. Maintained custom crawl rate of 0.1/sec for www.example.com. Will review this once things have settled.


Thread source:: http://www.webmasterworld.com/robots_txt/4325733.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com