| Welcome to WebmasterWorld Guest from 188.8.131.52 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Scooter spidering root index pages|
Scooter is visiting all of my sites grabbing only root level index pages.
/robots.txt HTTP/1.0" 302 209 "-" "Scooter-3.0"
/ HTTP/1.0" 200 3802 "-" "Scooter-3.0"
/index.html HTTP/1.0" 200 3802 "-" "Scooter-3.0"
The funny thing is that it is grabbing the server-specified root page AND grabbing the index.html page. I never submitted nor linked any of these sites by index.html.
Anyone else seeing the same?
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved