Welcome to WebmasterWorld Guest from 54.166.220.138

Forum Moderators: open

Message Too Old, No Replies

New crawler version using cookies?

Yahoo Vertical Crawler

     
11:43 am on Mar 5, 2004 (gmt 0)

10+ Year Member



Hi,

Is this a new crawler?

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

This crawler uses cookies.

The old FAST-WebCrawler/3.8 is also visiting my site at the same time AFAIK that one doesn't use cookies.

4:59 pm on Mar 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yep, it looks new to me, and yep, it uses cookies. Unfortunately, it doesn't seem to use robots.txt.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month