Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
interesting thing is that the site was down for 30 minutes just before the bot showed up... hope that doesn't hurt anything...
it also didn't ask for robots.txt, I guess its using a cached version from freshbot which was on the site 12 hours earlier?
[edited by: Stefan at 3:35 pm (utc) on Feb. 6, 2003]
Requested one page so far today (2/6) at 8:39am. Not a sign since.
This is a PR5 site with currently over 50k pages in the index. This makes me nervous.
joined:Oct 23, 2002
<added>Webmasterworld rocks too.</added>
18.104.22.168 - - [05/Feb/2003:18:50:39 -0800] "GET /robots.txt HTTP/1.0" 404 645 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
22.214.171.124 - - [05/Feb/2003:18:50:39 -0800] "GET / HTTP/1.0" 200 65291 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
About how long does it spend doing deep crawling?
It will show a log entry for every page linked by the time it's done. It might crawl through a few times.