Yahoo-VerticalCrawler-FormerWebCrawler

new crawler agent

[Robert Thivierge]

I was wandering if anybody else has seen this agent in their logs. In my case it took 100+ files, including several that were excluded in robots. Previously FAST/AllTheWeb's crawler's were well behaved.

The ip resolves to overture.com (not fastsearch.net).

It gives a url for info, which is the same as the one for the old "fast" agent. But, previously Fast/AllTheWeb seemed to respect my robots.txt files.

No big deal, but I'm just curious if anybody had a problem with this, or can point to a Yahoo/Overture/AllTheWeb page specifically about it. I won't be banning it (for now) since I want to be in AllTheWeb.

rob

[bull]

Can you provide a IP address?

[Robert Thivierge]

ip=66.77.73.32=shop-gw.sac.overture.com

[a_chameleon]

I can't resolve the entire IP, but 66.77.73.32 resolves

Qwest Cybercenters QWEST-CYBERCENTER-2 (NET-66-77-0-0-1)
66.77.0.0 - 66.77.255.255
Fast Search, Inc. QWEST-MCC-FASTSRCH3 (NET-66-77-73-0-1)
66.77.73.0 - 66.77.73.255
Name: shop-gw.sac.overture.com
IP Address: 66.77.73.32
Location: Sacramento (38.583N, 121.494W)
Network: Unknown

in NeoTrace.

[pendanticist]

No big deal, but I'm just curious if anybody had a problem with this, or can point to a Yahoo/Overture/AllTheWeb page specifically about it. I won't be banning it (for now) since I want to be in AllTheWeb.

[webmasterworld.com...]