ExoticCrawler

ban it?

[guitaristinus]

193.29.77.221 - - [01/Nov/2004:06:28:51 -0500] "GET /robots.txt HTTP/1.0" 200 93 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows XP) ExoticCrawler"

IP from Germany with no website. All it did was look at robots.txt.

All Google has about it are other people's logs. Interesting that search on Google shows ExoticCrawler always connected with "Mozilla/4.0 (compatible; MSIE 5.5; Windows XP)". Guess it's a MSIE 5.5 Windows XP thing. Another MSIECrawler?

[bull]

watch your logs from the same IP range (they do have a website, it is the national stock exchange):

Mozilla/4.7 [en] (X11; I; SunOS 5.8 sun4u)

checking for robots.txt I do not think it has anything to do with a real MSIE as "Windows XP" is not a valid Windows identifier for MSIE (it would be "Windows NT 5.1") Banned the whole range after the first issue.

[Matt Probert]

All it did was look at robots.txt.

That's rather unusual behaviour. Does your robots.txt file have any instructions to tell unknown spiders to leave?

Matt

[guitaristinus]

It actually came twice yesterday. From 193.29.77.220 as well as 193.29.77.221. Here's the other log entry:
193.29.77.220 - - [01/Nov/2004:06:02:12 -0500] "GET /robots.txt HTTP/1.0" 200 93 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows XP) ExoticCrawler"

Here's my robots.txt
User-Agent: NPBot
User-agent: psbot
User-Agent: vspider
User-agent: naver
Disallow: /

If I were to ban it it would be with htaccess using this:

RewriteCond %{HTTP_USER_AGENT} ^ExoticCrawler

But since it looked at robots.txt I could add it there.

Why would it just look at robots.txt if not going to spider the site? Why would it look at robots.txt if it is a bad bot?