| Welcome to WebmasterWorld Guest from 184.108.40.206 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Scooter spidering root index pages|
| 3:38 pm on Mar 22, 2001 (gmt 0)|
Scooter is visiting all of my sites grabbing only root level index pages.
/robots.txt HTTP/1.0" 302 209 "-" "Scooter-3.0"
/ HTTP/1.0" 200 3802 "-" "Scooter-3.0"
/index.html HTTP/1.0" 200 3802 "-" "Scooter-3.0"
The funny thing is that it is grabbing the server-specified root page AND grabbing the index.html page. I never submitted nor linked any of these sites by index.html.
Anyone else seeing the same?
| 1:14 pm on Mar 23, 2001 (gmt 0)|
Thanks for the heads up. I use the .htm, .html trick quite often when submitting to Alta.
I can see where scooter has been around to anything but what I've submitted Msgraph.
| 1:37 pm on Mar 23, 2001 (gmt 0)|
OOps forgot to add the IP addresses. These are the only ones I've seen but their could be more off the same C-Block
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved