Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Is slowing down Googlebot with a custom crawl rate a good idea?


jmccormac - 12:28 pm on Apr 17, 2012 (gmt 0)


It doesn't appear to be churning through links, serpsup.
Googlebot is crawling unique URLS. The problem for Googlebot is that the data here is on a narrow pipe so it can't download everything in one go. Though with the db behind the site at 162G, that might take some time.

Regards...jmcc
(Edit reason: typos - need to replace this keyboard.)


Thread source:: http://www.webmasterworld.com/google/4441847.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com