Openfind vs Google

Who spiders more...

[Lisa]

I am looking through log files and it appears that Openfind is just as agressive as Googlebot. I get spider almost daily by both but I am surprised at just how much Openfind can handle. I think openfind will be a rising star if they get there forumal down.

[Brett_Tabke]

The problem with OpenFind is they have been doing it for so long. They never had a url in their agent name so you couldn't even back track to find out who or what they were up to.

There were alot of us that banned them at the server door.

[Lisa]

I don't see a problem with them, anyone that wants to index my site is welcome to. Just as long as they follow the robot rules.

From what I see they must have most of the net now. All they need to do work on their search now.

[bobriggs]

Just as long as they follow the robot rules.

It's been messing around in my /cgi-bin/ for a while. I watched it fetch robots.txt and one page that had links into my /cgi-bin/ on July 18.

Yesterday it picked up all of the /cgi-bin/ links. It's totally ignoring my robots.txt

[Beachboy]

Brett, I don't know if you're aware of this, but I found this:

[openfind.com...]

...which is news to me. Until I saw this thread, I never heard of 'em. ;) They claim to be in Beta test, with 3.5-B. pages indexed.