Updated Crawler Info

Alltheweb / Yahoo

[MarkHutch]

Very busy this morning on one of our sites.

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

[WarmGlow]

MarkHutch wrote:

Very busy this morning on one of our sites.

. . . and requesting numerous files that are disallowed in the robots.txt file.

[MarkHutch]

I can't find them even requesting a robots.txt file in my logs.