Forum Moderators: open
/folder1/
Http Code: 200 Date: Jun 10 06:19:47 Http Version: HTTP/1.1 Size in Bytes: 2520
Referer: -
gent: EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx)
It even requested the style sheet.
Their website claims this: "We run a webbot as a part of our spam and web filtration services."
All well and fine, but why would they specifically crawl a directory that is disallowed from indexing but precious little of the rest of the site?
BTW, the whole site is quite innocuous - G rated - including the disallowed directory, it is only barred from indexing because we don't want people going straight to it without passing through the main site first.
Does anyone else have any experience with this bot?
It may also be one of those spiders/crawlers/whatevers that requires its own robots.txt entry -- an increasingly common, means-more-work trend (boo-hiss). I found this on an unrelated site's robots.txt page while Googling -- no clue if this is accurate or enough:
User-agent: EmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx)
Disallow: /
All I know for sure is that the primary "EmeraldShield" graphic's got a tpyo.
; )
The pages it tried to access indicated spider activity as web surfers would be unlikely to access those specific pages, but it got blocked automatically as an unknown user agent.
Still curious what the story is.
The user agent was the same as cited by Pfui in msg #:2.
It's a RIPE IP Address that belongs to Schlund + Partner AG in the 82.165.176.0 - 82.165.191.255 range. I'll post the specific IP Address if Dan says it's alright.
It read robots.txt, started out by taking a few files that weren't disallowed, then starting taking files from a folder that is disallowed and promptly fell into a spider trap.
I don't know about everyone else, but for me my list of allowed files would be much shorter than the list of disallowed files.
Also, as Jim suggested, by mentioning the disallowed files you're making it that much easier for some bot to find them.
Set up a dynamic robots.txt that shows only the allowed crawlers where they can go and show everything else a file that says basically "NO CRAWLING ALLOWED" and block them from accessing any pages. If they don't honor the robots.txt file that tells them to go away they are blocked from any pages with a whitelisted .htaccess file.
This instantly blocks everything except stealth crawlers, which is why I'm amused people still bother with maintaining blacklists.
incrediBILL: Is there info showing how to do that posted somewhere? I'm interested in implementing it.
[webmasterworld.com...]
Set up a dynamic robots.txt that shows only the allowed crawlers where they can go
1.) Bill's opt-out info via the preceding link is, as he explains more than once, an example, and NOT a server-ready, copy-paste thing.
(Here's hoping that someday he'll release either a program or a more specific how-to:)
2.) "Mozilla" in a UA does not automatically mean a good guy. There are lots of Mozilla-related (in name only) bots. Here [psychedelix.com] are more than a few (scroll down till you get to the Mozillas), and we report on still more all the time in this forum.
"Mozilla" in a UA does not automatically mean a good guy
Understatement of the decade.
I basically parse user agents as follows:
1. Let allowed search engines pass, check IP ranges to avoid spoofing.
2. Block anything that doesn't start with Opera or Mozilla
3. Scan everything else for a HUGE list of keywords that can appear anywhere in the user agent such as crawler, bot, http:, and about 100 others.
Sadly, this will also zap cellphones and PDA's unless you whitelist them in, but those people should have their hands on the wheel and not surfing the internet ;)
