| Updated Crawler Info Alltheweb / Yahoo |
MarkHutch

msg:845987 | 4:24 pm on Feb 28, 2004 (gmt 0) | Very busy this morning on one of our sites. | Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...] |
|
|
WarmGlow

msg:845988 | 6:00 pm on Feb 28, 2004 (gmt 0) | MarkHutch wrote: | Very busy this morning on one of our sites. |
| . . . and requesting numerous files that are disallowed in the robots.txt file.
|
MarkHutch

msg:845989 | 6:09 pm on Feb 28, 2004 (gmt 0) | I can't find them even requesting a robots.txt file in my logs.
|
|
|