Scooter has been spidering my site like mad the last few days. One think that wories me though is that it's generating heaps of 404s as it's trying to hit incomplete urls. The urls look like this: for example, it would try to go to www.mysite.bob/ne instead of www.mysite.bob/news.php It's also hitting urls that never existed, like www.mysite.bob/java while I'm sure my site never had a java folder.
I've seen similar reports from people, but I have never seen a answer to why this happens.
Hi driesie, This topic was discussed recently, but i've drawn a blank finding the thread. This search [google.com] has lots of mentions but neither Google or a sitesearch finds the thread. Anyone know the url?
In any case, I think it's a problem with the spider rather than your site, and it is safe to ignore.