EmeraldShield.com

robots rules don't apply to them

[koan]

Their spider found its way in my bot trap which is listed in my robots.txt. Here's their identification:

Hostname: secure.emeraldshield.com
User agent: EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx)

Curious, I checked their webbot.aspx page and found this little nugget:

Obey Robots.txt - It is not always in the best interest to obey robots.txt. Many sites containing questionable content will attempt to use this file to keep spiders/bots from finding it. The primary reason a site uses this file is to keep search engine spiders and such from attempting to index data they do not wish to be searched on or to keep them out of image folders. Our spiders are intellegent enough to not drain bandwidth from sites looking at these sites as we will not attempt to download just anything.

If they think robots rules don't apply to them, maybe they oughta find their way in your htaccess rules.

[Mokita]

If they think robots rules don't apply to them, maybe they oughta find their way in your htaccess rules.

I've barred them, and I think lots of others have too.

See this thread for more info:

[webmasterworld.com...]

[malachite]

Thanks koan. I came to see if there was anything here about this little fella and saw the snippet you posted. It's now on the banned list via .htaccess. :)