homepage Welcome to WebmasterWorld Guest from 54.198.185.156
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Sky IP requesting all files alphabetically
cyberdyne




msg:4309288
 3:24 pm on May 7, 2011 (gmt 0)

Has anyone else noticed this?
IP 2.220.166.34

Requests all files in the home directory in alphabetical order but doesn't stick to any robots rules.

 

Frank_Rizzo




msg:4309294
 3:26 pm on May 7, 2011 (gmt 0)

Could be a home user running something like Xenu on your site.

Do you have a sitemap file? Maybe that was read / processed in alphabetical order.

cyberdyne




msg:4309302
 3:42 pm on May 7, 2011 (gmt 0)

Yes I have a site map. I had a feeling it was something like that. Have blocked them for now, just to dissuade them.
Thanks

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved