User-Agent: OmniWeb

Disallowed

[guitaristinus]

Comes from a Mack browser. Works like MSIECrawler. It checked robots.txt before downloading site, so I added it to mine.

[pendanticist]

Uh, such might not be the case?

Omniweb + WebmasterWorld [google.com].

BjarneDM seems to have nailed it for GaryK in this thread [webmasterworld.com...] . In particular msg #:6. Even mentions the The Omni Group who produces the, uh, suspect item.

I DO have to wonder though if this browser doesn't have the same nasty capabilites as Alexa in it's ability to suck down a site quickly?

[edited by: volatilegx at 2:13 am (utc) on Nov. 28, 2004]
[edit reason] link removed [/edit]

[guitaristinus]

OmniWeb is the name of the agent that OmniWeb (the browser) uses to download a site for offline viewing. Someone can still use the browser even though its agent is disallowed from site.

It requested 8013 pages at about 3/second. No websites hosted using IP address.

Here's a line from log:

82.122.116.229 - - [22/Nov/2004:18:24:25 -0500] "GET /mypage/ HTTP/1.1" 200 7999 "http://mysite.com/" "OmniWeb"

[wilderness]

SetEnvIf User-Agent OmniWeb keep_out

will work