Welcome to WebmasterWorld Guest from

Forum Moderators: open

New crawler version using cookies?

Yahoo Vertical Crawler

11:43 am on Mar 5, 2004 (gmt 0)

10+ Year Member


Is this a new crawler?

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

This crawler uses cookies.

The old FAST-WebCrawler/3.8 is also visiting my site at the same time AFAIK that one doesn't use cookies.

4:59 pm on Mar 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Yep, it looks new to me, and yep, it uses cookies. Unfortunately, it doesn't seem to use robots.txt.

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month