Welcome to WebmasterWorld Guest from 54.198.185.204

Forum Moderators: open

Message Too Old, No Replies

New crawler version using cookies?

Yahoo Vertical Crawler

     
11:43 am on Mar 5, 2004 (gmt 0)

New User

10+ Year Member

joined:Feb 2, 2003
posts:27
votes: 0


Hi,

Is this a new crawler?

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

This crawler uses cookies.

The old FAST-WebCrawler/3.8 is also visiting my site at the same time AFAIK that one doesn't use cookies.

4:59 pm on Mar 19, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Yep, it looks new to me, and yep, it uses cookies. Unfortunately, it doesn't seem to use robots.txt.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members