Single hit on sitemap.xml

[Frank_Rizzo]

I don't have a sitemap but daily I get requests for sitemap.xml. This is not from a SE though - it is always from a home user from some of the main UK isps.

There is no other log entry for that IP either. It is a simple request for the sitemap and nothing else:

86.nnn.nnn.nnn - - [04/Mar/2009:11:07:53 +0000] "GET /sitemap.xml HTTP/1.1" 404 917 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6" 0 widgets.co.uk "-" "-"

The IPs are not from rogue countries. They are all from main UK isps.

Why is this happening? Are they part of botnets where this IPs sole task is one of reconnaissance?

[tangor]

Sometimes there are Those Who Search. One-offs I don't worry about. One-offs multiplied by several hundred/thousand, on the other hand, do get my attention. Or, it might be a fellow looking to rip the site with ease by having a roadmap to all the goodies. :)

[enigma1]

Yes I see these requests for sitemap.xml daily. They're blocked and the bots don't even know it. I've setup the sites to keep sending them redirect headers, continuously bouncing them across different domains. I then run scripts against the server logs to see their patterns and hops as they also follow the redirects using different IPs so I have to track them via distinct link strings. I think lame requests need lame responses.