Welcome to WebmasterWorld Guest from 54.197.116.116

Forum Moderators: open

Message Too Old, No Replies

New crawler version using cookies?

Yahoo Vertical Crawler

   
11:43 am on Mar 5, 2004 (gmt 0)

10+ Year Member



Hi,

Is this a new crawler?

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

This crawler uses cookies.

The old FAST-WebCrawler/3.8 is also visiting my site at the same time AFAIK that one doesn't use cookies.

4:59 pm on Mar 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yep, it looks new to me, and yep, it uses cookies. Unfortunately, it doesn't seem to use robots.txt.