Welcome to WebmasterWorld Guest from 54.226.194.180

Forum Moderators: open

Message Too Old, No Replies

Scooter spidering root index pages

     

msgraph

3:38 pm on Mar 22, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Scooter is visiting all of my sites grabbing only root level index pages.

/robots.txt HTTP/1.0" 302 209 "-" "Scooter-3.0"

/ HTTP/1.0" 200 3802 "-" "Scooter-3.0"

/index.html HTTP/1.0" 200 3802 "-" "Scooter-3.0"

The funny thing is that it is grabbing the server-specified root page AND grabbing the index.html page. I never submitted nor linked any of these sites by index.html.

Anyone else seeing the same?

Brett_Tabke

1:14 pm on Mar 23, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Thanks for the heads up. I use the .htm, .html trick quite often when submitting to Alta.
I can see where scooter has been around to anything but what I've submitted Msgraph.

msgraph

1:37 pm on Mar 23, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



OOps forgot to add the IP addresses. These are the only ones I've seen but their could be more off the same C-Block

209.73.164.132
209.73.164.127

 

Featured Threads

Hot Threads This Week

Hot Threads This Month