Normally the freshbot, 64.68.82.xx, just grabs a few pages off of my sites, but today it is acting like the deepcrawler and grabing everything. As much as 1200 pages on one site. I don't think I have ever seen this before. Is this like last month where a mid-month crawl became more or less permanent? I can't remember if it was the 64 or 216 bot last month. Does anyone have any thoughts on this?
wired4fun
5:23 am on Feb 17, 2003 (gmt 0)
I'm confused as well. Freshbot usually grabs a thousand or so pages per day... and just picked up over 15,000. Looks like google is trying to make many more results minty fresh :-)
Jimmie
5:45 am on Feb 17, 2003 (gmt 0)
freshbot is doing a deep crawl on my site too right now, but interestingly (216*) dropped by again yesterday after deepcrawling my site around the 7th.
NickCoons
6:05 am on Feb 17, 2003 (gmt 0)
Freshbot made about 600 requests for pages on my site today, and at last count I only had about 400 pages.
I didn't check to see which pages it's requesting more than once, but I've also noticed how odd it is to have Freshbot looking at so much of my content.
