Page is a not externally linkable
-- Google SEO News and Discussion
---- Mozilla Googlebot Crawling Deep and Fast
AlexK - 12:55 am on Feb 15, 2006 (gmt 0)Which G-Bot? ('normal' G-Bot, Mozilla-Bot (M-Bot), Adsense-Bot (A-Bot), Image-Bot (I-Bot)) (see also below)
Starting to look interesting, is it not?
Some more detail would be useful. eg
Dynamic or HTML site?
How deep? (number of pages does not really help, and can look like bragging (!); so, "an 80% increase in pages taken" would be more helpful in that context)
Do not forget dates
Discriminating between G-Bots: If you use AWStats, making the following changes to robots.pm will give individual stats for the different bots:
It will then be necessary to remove that month's db file (assuming that your raw logfiles contain full stats for that month).
|# 2005-06-25 googlebot changed to ^googlebot\/ + googlebot added to RobotsSearchIDOrder_list1 |
# + to distinguish between HTTP/1.0 (former, old) and HTTP/1.1 (new, Mozilla/5.0)
# + bots (different beasts)
'^googlebot\/', # must be before googlebot
'googlebot\-image', # must be before googlebot
'googlebot\-mobile', # must be before googlebot
'^googlebot\/','Googlebot HTTP/1.0 (google.com/bot.html)',
'googlebot\-mobile','Googlebot-Mobile (Nokia6820 google.com/bot.html)',
'googlebot','Googlebot HTTP/1.1 (Mozilla/5.0 google.com/bot.html)',
Dynamic sites: It is at times like this that PHP & other sites miss the Content-Negotiation provided as standard by webserver software for static HTML pages (in practical terms, an unchanged dynamically-produced page appears brand-new, and the bots re-request it). If a PHP-site, have a look at this thread for a Content-Negotiation Class [webmasterworld.com]. Easy to implement, and will reduce bandwidth + server-load. Some support on the Class is also available on this and following pages [webmasterworld.com].
Thread source:: http://www.webmasterworld.com/google/33086.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com