Disposable search engines?

which ones

[martin]

I have some stupid requests (404s) from Openfind data gatherer.

Is it enough to ban it, I have not referrers from it and I don't know anybody using it?

/add
Are there any others that are useless enough to ban them.

[Nick_W]

I'd leave it. It seems fairly polite and they may just gain ground sometime in the future...

Nick

[martin]

Oh, did I mentioned they did 29 requests for non-existing files, that never existed actually.

It seems that they try to gather more data than exists ;-) I haven't banned them but if it happens again...

Just wanted to know if somebody else has had problems with these guys.

[NFFC]

I have them down as the next big thing in the search engine world, I wouldn't ban them at this stage.

[Nick_W]

Yes, from the little I've read (mainly here) they seem to be well funded with a 'go get 'em' attitude...

Nick

[ritch_b]

We had OpenFind visit early last week and request a large number of pages which don't exist and have never existed.

Didn't seem particularly agressive though, so I'm giving this particular spider the benefit of the doubt.

[martin]

>I have them down as the next big thing in the search engine world, I wouldn't ban them at this stage.

OK, I'll wait if they do more wrong stuff.

PS. I found that Google also would try to crawl non-existent pages, I submitted something and they tried to fetch it a few days later.

[martin]

They just tried to fetch 35 more non-existant pages.

It looks to me that they probably have messed up their DNS cache because this never existed on my site. All URIs start with /italy/