Same UA, different IPs, a lot of 404s

[smallcompany]

I noticed a lot of 404s caused by this user agent:

User Agent = Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727)

Sometimes it's 15 errors in a minute, sometimes it's 3 or 4.

IPs are being switched and seem US based.

The pattern is the same. It's like on the page, and then it looks for a URL from main navigation, but instead of going back to the root, it applies it to the current page which is two or three levels down. That causes an avalanche effect as it keeps looking for those broken links out of the blue.

I have this trouble with UAs like AddThis.com as well, but this above is suspicious as it looks like pretending to be a human while it obviously is not. What it could be?

I don't have such trouble with legitimate bots, so I assume this is not that my site somehow serves broken links.

Thanks

[keyplyr]

Do a WHOIS look-up on the IP addresses. If they all belong to just a couple server farm (hosting) companies it is most likely one actor looking for open doors at your server in addition to following page links.

If the IP addresses belong to a mixture of server companies and ISPs it may be a botnet of infected accounts purposed to do mischief, attempting to access a laundry list of vulnerable documents (whether these files exist on your server or not.)

This is not unique. Read-up on some of the posts here. One thing is for sure... if it can be done, someone will attempt to do it.

[wilderness]

I noticed a lot of 404s caused by this user agent:

User Agent = Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727)

If you've confirmed that they are all the precisely same UA?
The following in a mod_rewrite line will work and not require any IP lookups and/or denies:
Trident/4\.0;\ \.NET CLR\ 2\.0\.50727\)$

[keyplyr]

Bad idea to recommend code blocking access to someone else's business if it is not you that will experience the result to see exactly who is getting blocked.

[tangor]

If your result is a 404 then I wouldn't worry about it ... unless you are running a custom 404 of significant kbs. In any case, Ignore these as a cost of doing business, while checking to make sure there's no real danger to your site.

If you want to control this behavior then work from the requesting side (what are they asking for) and send them a 403 which will make it easier to filter out of your log reports. ie. on a site that does not use php for anything all requests for php are 403 ... usually a much smaller file! If blocking by UA make sure it is not TOO broad matched you'll take out live humans! If by IP learn about it as it is too easy to nuke millions in a single line (unless that is intended!).

[wilderness]

I noticed a lot of 404s caused by this user agent:

User Agent = Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727)

Bad idea to recommend code blocking access to someone else's business if it is not you that will experience the result to see exactly who is getting blocked.

FWIW, if he's looking at 404's and repeated UA's than he'll surely see the result of any rewrite in place.