| New crawler version using cookies?
Yahoo Vertical Crawler |
misja
msg:221032 | 11:43 am on Mar 5, 2004 (gmt 0) |
Hi, Is this a new crawler? Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...] This crawler uses cookies. The old FAST-WebCrawler/3.8 is also visiting my site at the same time AFAIK that one doesn't use cookies.
|
mcavic
msg:221033 | 4:59 pm on Mar 19, 2004 (gmt 0) |
Yep, it looks new to me, and yep, it uses cookies. Unfortunately, it doesn't seem to use robots.txt.
|
|
|
