Welcome to WebmasterWorld Guest from 54.224.77.47

Forum Moderators: martinibuster

Message Too Old, No Replies

Updated Crawler Info

Alltheweb / Yahoo

     
4:24 pm on Feb 28, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:742
votes: 0


Very busy this morning on one of our sites.

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]
6:00 pm on Feb 28, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 23, 2003
posts:77
votes: 0


MarkHutch wrote:
Very busy this morning on one of our sites.

. . . and requesting numerous files that are disallowed in the robots.txt file.

6:09 pm on Feb 28, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:742
votes: 0


I can't find them even requesting a robots.txt file in my logs.