homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
scooter spiders incomplete urls

 10:08 am on May 27, 2003 (gmt 0)

Scooter has been spidering my site like mad the last few days.
One think that wories me though is that it's generating heaps of 404s as it's trying to hit incomplete urls.
The urls look like this:
for example, it would try to go to www.mysite.bob/ne
instead of www.mysite.bob/news.php
It's also hitting urls that never existed, like www.mysite.bob/java
while I'm sure my site never had a java folder.

I've seen similar reports from people, but I have never seen a answer to why this happens.

Does anybody have any ideas?




 7:11 pm on May 27, 2003 (gmt 0)

Hi driesie,
This topic was discussed recently, but i've drawn a blank finding the thread. This search [google.com] has lots of mentions but neither Google or a sitesearch finds the thread. Anyone know the url?

In any case, I think it's a problem with the spider rather than your site, and it is safe to ignore.


 9:24 pm on May 27, 2003 (gmt 0)

Thanks for the answer.
I have searched as well, but could only find this post:

Where mvl22 describes the same problem, but there's no answer to what it could be.

Glad to hear it's not my site though.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved