| Updated Crawler Info
Alltheweb / Yahoo |
MarkHutch
msg:845987 | 4:24 pm on Feb 28, 2004 (gmt 0) |
Very busy this morning on one of our sites. | Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...] |
|
|
WarmGlow
msg:845988 | 6:00 pm on Feb 28, 2004 (gmt 0) |
MarkHutch wrote:
| Very busy this morning on one of our sites. |
|
. . . and requesting numerous files that are disallowed in the robots.txt file.
|
MarkHutch
msg:845989 | 6:09 pm on Feb 28, 2004 (gmt 0) |
I can't find them even requesting a robots.txt file in my logs.
|
|
|
