joined:May 20, 2003
I just checked my logs and saw that Jeeves is spidering a whole bunch of pages that don't exist and never existed. What it's doing is taking a valid file name and adding a space and a 1 to it, something like "/filename.html 1" but it also took "/ 4" and "/ 19" which don't exist at all. It had no problem getting my robots.txt file though. Any reason why this would happen?