Hotbot-Lycos Robot

[jdancing]

Over the past month the Hotbot-Lycos Robot has been my hardest hitting robot. When I go to HotBot, their results are similar to google's.

Is the Hotbot-Lycos Robot just another Google spider now? It's hard to believe a spider that hits my site so hard is from a search engine that brings basically zero traffic.

[wilderness]

Any bot has an appetite which requires nourishment.

If the visitor returns for the spidering are not rewarding to you, and only you are capable of making that decision, than either deny the bot or disreagrd the log lines and traffic.

There's not a middle option, unless you exclude directories and/or portions of your website.

Might part of the problem be with images?
Do you have your images in an image only folder which is excluded from crawling in your robots.txt?

[jdancing]

I don't mind the spidering, just wondering why a second tier search engine would spider my site so much.

[wilderness]

jdancing,
The content of many of my pages is most interesting to many of Euro countries (Denmark, Sweden, Norway and a few others.) The horsey folks there, on the average, are more knowledgable concerning old breeding and history than their North American counterparts. The problem is that I stand no way to benefit from that traffic. They surely aren't going to advertise on a NA website.
As a result, the bots from this portion of RIPE just keep trying and trying, over and again.
The traffic is just a drain on resources and since I neither read or speak a non-English language, add also the fact tha most online translators are not really effective, how would I determine if any of my content had been copied? The efforts required by me would just be too time consuming.
Add to this, the time difference between NA and Europe and most of these vsitors and bots trafficing while I was sleeping. Since I do my monitoring and determinations manually, their traffic was thorn.

In the end, even though it was not my desire, denying access to the RIPE visitors was the simpliest solution.

Ask Jeeves visits me weekly and crawls my entire site. No images, as they are excluded in robots. The return of traffic I get from Jeeves (or any SE of similar size or less) is hardly (IMO) allowing them to crawl. I do however allow Jeeves.

For me there are other issuse as well.
In this recent submission:
[webmasterworld.com...]
I mentioned an integrity issue.
BTW, this bot visited yesterday after an absence and appeared with a new UA:

206.123.66.243 - - [03/Dec/2004:19:15:36 -0800] "GET /robots.txt HTTP/1.0"
200 2957 "-" "SportsBot 1.0 h**p://Search4Sports.net/"

FAST at one time was crawling me regularly with hardly any return traffic.
The IBM Almaden would crawl my entire site if allowed and I stand nothing to gain from that as their primary intent is in serving their customers in a 3rd party capacity without condsidering my compensation or efforts.
There are many more examples which have been discussed in this forum time and again.

Bottom line is that we each must decide what is beneficial or detrimental in both our visitors and bots.
We each have a different scale of meausremnt as well ;)

Don