homepage Welcome to WebmasterWorld Guest from 54.196.206.80
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Sky IP requesting all files alphabetically
cyberdyne




msg:4309288
 3:24 pm on May 7, 2011 (gmt 0)

Has anyone else noticed this?
IP 2.220.166.34

Requests all files in the home directory in alphabetical order but doesn't stick to any robots rules.

 

Frank_Rizzo




msg:4309294
 3:26 pm on May 7, 2011 (gmt 0)

Could be a home user running something like Xenu on your site.

Do you have a sitemap file? Maybe that was read / processed in alphabetical order.

cyberdyne




msg:4309302
 3:42 pm on May 7, 2011 (gmt 0)

Yes I have a site map. I had a feeling it was something like that. Have blocked them for now, just to dissuade them.
Thanks

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved