How would I findout the way google has crawled my site?, How deep and What pages it missed out and so on.
How do you guys go about doing so?
Regards
jpjones
1:47 pm on Feb 25, 2003 (gmt 0)
Hi Martin,
Look at the raw logs for the site, looking for any requests by the user agent "Googlebot/2.1" - this is Google's bot.
You can then match up the requests to your site pages and work out what has been crawled and what hasn't.
JP
jpjones
1:52 pm on Feb 25, 2003 (gmt 0)
Just to add to the above - you can also recognise googlebot by it's IP Address.
Look at this forum [webmasterworld.com] for further information.
JP
WebRankInfo
7:28 pm on Mar 15, 2003 (gmt 0)
for those of you with PHP support, you may try GoogleStats. It is a free PHP application detecting the visits of search engines robots, and presenting statistics in tables and graphics. It is available in french, english, german and spanish. htpp://www.googlestats.com/english/