Welcome to WebmasterWorld Guest from 188.8.131.52
Other visitors that were on the site around that time only have one or two hits and they do not include my reviews section. Notice the 301 hits. Among those 301 hits are links like this:
Has anyone seen this type of activity before from a bot or crawler?
[edited by: volatilegx at 9:16 pm (utc) on June 15, 2005]
[edit reason] removed specifics [/edit]
Hi Jabba and welcome to WebmasterWorld happy!
I haven't seen this type of behaviour from the Convera Crawler before, but this is very interesting.
If more info is needed I will try to provide it.
Trojan or virus? Don't know but I do know these guys are new and they initially brought up the suggestion that they could have a malicious script embedded in the crawler code.
Looks like just another maggot until proven otherwise. I've emailed the contact address for an explanation. But I don't buy that hijack theory.
My gripe is "the why" of allowing them to utilize private resources (websites and bandwidth)without expecting compensation from either the "so called scraper" or their customers?
After all they are collecting the data to be utilized in a non-internet capacity. More like an intranet.
I feel the same way about univerities. And I do realize that much research (such as google and other projects) begins at universities. However, they have vaild resources in the way of grants with paid staff (professors) and students doing the majority of the work to further their career beyond the data they mine from privately owned web sites.
Another good example is Archive Org. It's an excellent resource and concept. The moog point, IMO, is that they will sell terabytes of collected data to anybody that wants to pay.
That payment concept is not under the theme of what most webmasters allow their site to be spidered.
The term "third party" is very broad and entials many companies not offering search engines. IBM Almaden is another example that only collects data to display in a closed enviroment to paid customers.
I have no desire to allow these types of bots or software's in my sites. UNLESS they are willing to send some compensation my way.
Of course they'd have to change their entire concept of doing business before that would happen ;)