homepage Welcome to WebmasterWorld Guest from 54.197.171.109
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Scooter spidering root index pages
msgraph




msg:20936
 3:38 pm on Mar 22, 2001 (gmt 0)

Scooter is visiting all of my sites grabbing only root level index pages.

/robots.txt HTTP/1.0" 302 209 "-" "Scooter-3.0"

/ HTTP/1.0" 200 3802 "-" "Scooter-3.0"

/index.html HTTP/1.0" 200 3802 "-" "Scooter-3.0"

The funny thing is that it is grabbing the server-specified root page AND grabbing the index.html page. I never submitted nor linked any of these sites by index.html.

Anyone else seeing the same?

 

Brett_Tabke




msg:20937
 1:14 pm on Mar 23, 2001 (gmt 0)

Thanks for the heads up. I use the .htm, .html trick quite often when submitting to Alta.
I can see where scooter has been around to anything but what I've submitted Msgraph.

msgraph




msg:20938
 1:37 pm on Mar 23, 2001 (gmt 0)

OOps forgot to add the IP addresses. These are the only ones I've seen but their could be more off the same C-Block

209.73.164.132
209.73.164.127

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved