Welcome to WebmasterWorld Guest from 54.144.79.200

Forum Moderators: martinibuster

Message Too Old, No Replies

Updated Crawler Info

Alltheweb / Yahoo

     

MarkHutch

4:24 pm on Feb 28, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Very busy this morning on one of our sites.

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

WarmGlow

6:00 pm on Feb 28, 2004 (gmt 0)

10+ Year Member



MarkHutch wrote:
Very busy this morning on one of our sites.

. . . and requesting numerous files that are disallowed in the robots.txt file.

MarkHutch

6:09 pm on Feb 28, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I can't find them even requesting a robots.txt file in my logs.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month