Welcome to WebmasterWorld Guest from 54.224.230.193

Forum Moderators: open

Message Too Old, No Replies

New crawler version using cookies?

Yahoo Vertical Crawler

     
11:43 am on Mar 5, 2004 (gmt 0)

New User

10+ Year Member

joined:Feb 2, 2003
posts:27
votes: 0


Hi,

Is this a new crawler?

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

This crawler uses cookies.

The old FAST-WebCrawler/3.8 is also visiting my site at the same time AFAIK that one doesn't use cookies.

4:59 pm on Mar 19, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Yep, it looks new to me, and yep, it uses cookies. Unfortunately, it doesn't seem to use robots.txt.