Welcome to WebmasterWorld Guest from 50.16.24.12

Forum Moderators: open

New crawler version using cookies?

Yahoo Vertical Crawler

   
11:43 am on Mar 5, 2004 (gmt 0)

10+ Year Member



Hi,

Is this a new crawler?

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

This crawler uses cookies.

The old FAST-WebCrawler/3.8 is also visiting my site at the same time AFAIK that one doesn't use cookies.

4:59 pm on Mar 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yep, it looks new to me, and yep, it uses cookies. Unfortunately, it doesn't seem to use robots.txt.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month