Welcome to WebmasterWorld Guest from 54.160.221.82

Forum Moderators: open

Message Too Old, No Replies

Scooter spidering root index pages

     
3:38 pm on Mar 22, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 29, 2000
posts:1425
votes: 0


Scooter is visiting all of my sites grabbing only root level index pages.

/robots.txt HTTP/1.0" 302 209 "-" "Scooter-3.0"

/ HTTP/1.0" 200 3802 "-" "Scooter-3.0"

/index.html HTTP/1.0" 200 3802 "-" "Scooter-3.0"

The funny thing is that it is grabbing the server-specified root page AND grabbing the index.html page. I never submitted nor linked any of these sites by index.html.

Anyone else seeing the same?

1:14 pm on Mar 23, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Thanks for the heads up. I use the .htm, .html trick quite often when submitting to Alta.
I can see where scooter has been around to anything but what I've submitted Msgraph.
1:37 pm on Mar 23, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 29, 2000
posts:1425
votes: 0


OOps forgot to add the IP addresses. These are the only ones I've seen but their could be more off the same C-Block

209.73.164.132
209.73.164.127