stat [e-mail]

just noticed this today...

[BillyS]

I just happened to notice this today in my logs:

66.92.*.* - - [02/Oct/2004:08:31:18 -0400] "GET /robots.txt HTTP/1.0" 200 484 "-" "stat ([e-mail])"
66.92.*.* - - [02/Oct/2004:08:31:19 -0400] "GET / HTTP/1.0" 200 19826 "-" "stat [e-mail]"

Could not find anything here or on the web yet. Anyone else see this?

[edited by: Brett_Tabke at 1:36 pm (utc) on Oct. 2, 2004]

[Brett_Tabke]

someone running a bot from home with a fake agent name.

[bull]

Use an anti-lowercase rule to get rid of many of these rogue ones

RewriteCond %{HTTP_USER_AGENT} ^[a-z0-9]+
RewriteCond %{HTTP_USER_AGENT}!^msnbot
RewriteCond %{HTTP_USER_AGENT}!^contype
RewriteRule!robots\.txt - [F]

[uncle_bob]

Visited me yesterday. Odd the UA is different for robots.txt and / requests. I'm never keen on spiders that don't give a url in their user-agent.

[bull]

Odd the UA is different for robots.txt and / requests

Rogue Yahoo bots [webmasterworld.com] do the same ;-)

[aerostat]

stat is an experimental crawler for a next generation search engine like service, which would try to be webmaster friendly. It's interesting that it only got noticed by now after it has crawled tens of millions of pages. I haven't seen any complains (aside from a couple of curious inquiries) yet regarding it's behavior.

If you think it's misbehaving on your site, send a note to [e-mail] and it'll be dealt with.

Thanks

[wilderness]

It's interesting that it only got noticed by now after it has crawled tens of millions of pages.

When did it begin identiying itself with a UA?

The should provide an insight into how many webmasters view their logs and also take steps to assure the protection of their data.

Bots and/or bot creators have been crawling for such a length of time that most have little regard for websites or webmasters desires.

[aerostat]

It's been using the same ua id from the beginning of the current crawl, since early september. It adheres to the robots.txt instructions. The current minimum page fetch interval for a site is 30 seconds (there is no scheduled delay between the fetch of robots.txt and the first page.) I suspect most webmasters won't mind as there are zero complaints so far. The intention is to create a service that is mutually beneficial to the sites crawled and the service provider.

BTW, we're not affiliated with either aerostat.com or aerostat.net.

Thanks.