When I checked my stats today, I could not believe what I saw: Scooter hat indexed a huge amount of pages. But happyness turned into astonishment when I had a closer look at my log files:
Scooter had indexed just yesterday thousands of urls 9 levels and more deep, containing about 300 to 400 characters!
Due to a leak in my URl rewriting code it got href="widget-name/manufacturer/" instead of href="/...". The page containing the error was not reachable by a link (probably the reason why I haven't noticed the error before), thus Scooter must have dropped the "manufacturer" and gone to /widget-name instead.
Glad I checked my stats today. I don't know where this would have ended (Scooter going down the infinite pyramide)
Now I am anxious to see, how I rank! Hope I don't get banned ;-)
from what I see, this bot is really buggy, it's requesting uri that do not exist on my server, does not respect absolute paths (like href="/something") and sends uncomplete uri like "login.ph" instead of "login.php"
Scooter produces tons of 404 everytime it come by my sites whereas all my links are valid
Scooter produces heaps of 404s on my site as well. Like mentioned before it seems to chop of urls all the time. It also seems to 'try' to connect to anything/java Very strange, as I don't have anything "java" on my site. I get an email everytime a 404 is produced, and each time I get scootered, I have to crawl through a heap of mails!