www.go2-yellowpages.com

Deep crawling for awhile now, and incorrectly

[savvy1]

www.go2-yellowpages.com - - [23/Sep/2002:08:44:38 -0400] "GET /green-widgets//blue-widgets.html HTTP/1.0" 200 39523 "-" "-"
No UA and No referrer set.

Anyone have any info on this one? A search (of Wbmwrld) turned up nothing. Been getting deep crawled by it, and its not even crawling correctly.

I have some links from /index.html to /blue-widgets.html and its incorrectly crawling them as /green-widgets//blue-widgets.html, stupid bot.

I visited the address that IP reverses to (www.go2...) and it redirects to www.only-yellow-pages.com which appears to be a yp/web search engine. Don't see any info about the crawler. Is the data from their own crawling or do they get fill from elsewhere? Is this bot worth letting crawl my site, especially considering its not even doing it correctly?:)

[savvy1]

nobody even seen this bot in their logs?

[jeremy goodrich]

I've never seen it in any logs I monitor...however, I do get every day or couple days a random user agent that's new to me.

First thing I do is search for it...if there's not much to be found on google, wisenut, teoma, alltheweb, then I usually just ignore it.

Also, if the bot is crawling 'incorrectly' then that might be why nobody else has seen it - you could be the bot owners first trial run :)

[savvy1]

Interesting. It grabbed 1100 pages from Sep 17 - Sep 24.. I guess I should say made 1100 requests as it screwed up most of them...