Sky IP requesting all files alphabetically - Sitemaps, Meta Data, and robots.txt forum at WebmasterWorld - WebmasterWorld

Forum Moderators: goodroi

Message Too Old, No Replies

Sky IP requesting all files alphabetically

cyberdyne

3:24 pm on May 7, 2011 (gmt 0)

10+ Year Member

Has anyone else noticed this?
IP 2.220.166.34

Requests all files in the home directory in alphabetical order but doesn't stick to any robots rules.

Frank_Rizzo

3:26 pm on May 7, 2011 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Could be a home user running something like Xenu on your site.

Do you have a sitemap file? Maybe that was read / processed in alphabetical order.

cyberdyne

3:42 pm on May 7, 2011 (gmt 0)

10+ Year Member

Yes I have a site map. I had a feeling it was something like that. Have blocked them for now, just to dissuade them.
Thanks