Netscape 3 UA, and Favicon.ico

Bad Bot! Bad Bot!

[Lisa]

Ok, I have this evil bot on my site using the user agent Mozilla/3.01 (compatible;) and I have blocked the little guy, but now all he does is sit there and request the favicon.ico from me. What is up with that?

I am getting lots of evil bots from apnic, So I just have to question anything coming from Asia now.

[idiotgirl]

Are you sure it's a bot? Lots of UA's and proxies use Mozilla/3.01. And if that's a real viewer, they may have hit favicon.ico just to bookmark your site. It might be innocent... unless, of course, there are no page views to go with it! That would be a heads-up right there.

The decision to ban all Asian sites might be a tough call, though it seems 95+% of what comes from there is all spambots and harvesters. But unless your Moz/3 hits are all from Asian IP blocks - you might check a little further before giving them the boot.

[Lisa]

Requesting the favicon.ico over 60 times is no human. They had lots of hits on my site, but I cut off all those pages. Now it just sits there requesting favicon.ico. I have no idea what the thing is doing, but it is wierd.

[idiotgirl]

What's the IP address?

[Lisa]

202.149.208.73 - - [09/Jul/2002:23:52:56 -0700] "GET /favicon.ico HTTP/1.0" 404 578 "-" "Mozilla/3.01 (compatible;)"

[idiotgirl]

Oh heck yeah. I banned the whole block. But I haven't seen that behavior before. Must be caught in a loop. I just had some recently looping requests for index.cgi, homepage.cgi, forrmail.cgi. I banned 202-anything.

And no more BaiDuSpider, either.

[Lisa]

Well, I count 2 real users in the 202.x.x.x and like 4 bots in the last 24 hours. So I don't want to shut them all off. But it is very tempting.

[bird]

I wouldn't block "Mozilla/3.01 (compatible;)" quite as rapidly. Looking at my log files, this is most likely a proxy server that happens to be installed at many locations. The typical pattern is for the *.html requests to show the UA of the requesting browser, while image and other media requests will use the above generic UA.

If you block this one, then lots of users will only see your HTML but no images. And if a browser couldn't fetch your favicon on the first time, it will rerequest it each time the user retreives your page from his browser cache. If you block the IP range, then you're likely to lose many legitimate visitors. Note that the UA sometimes also comes without the semicolon.