Omni-explorer

Hammered my site a little earlier

[Stefan]

I tried a G search of WW on this but only found threads from last Spring.

This thing ripped through my site an hour ago:

OmniExplorer_Bot/4.32

It did ask for robots.txt first, but it looks like it took all 350+ pages in the course of about 15 min's. I don't like seeing bots come out of nowhere, with limited explanation, and suddenly tear through my site and copy every html file. The website gives not much more than a "Coming soon" page.

Anyone have anymore info on these guys?

[chirp]

I've been blocking them for months:

RewriteCond %{HTTP_USER_AGENT} ^OmniExplorer 
RewriteRule .* - [F]

but they say they're better behaved now: [omni-explorer.com...]

[Stefan]

Thanks, Chirp.

Yeah, I think I'll lock them out. If they can't even explain what they're going to do with the stuff, they're not welcome.

[fiestagirl]

There have been a few discusssions on this.
[google.com...]

[Stefan]

Thanks Fiestagirl. Yeah I found those before I started the thread. I was wondering if there was any recent info on whether it's a scraper or a future SE. The posts from last Spring didn't really indicate if anyone knew what their intentions were. Anyway, whatever they are, they have all my content up to a couple of days ago. I've added a few pages since then, so they're beat on those ;-)

One thing about that crawl, by checking the hits in the logs, I now know that I have 373 pages on the main site (plus a few more as of today). I wasn't really sure - thought it was about 350 ;-)