Just found Scooter/3.3.vscooter grabbing a bunch of files disallowed to all ("*") in my robots.txt file. The only thing I can think of is that the directories were named things like img/ foo/images/
and I used a trailing slash. I saw here that apparently some browsers misinterpret that? Removing the slash somewhat bothers me, because what's stopping some *other* stupid robot from deciding that "foo/images" is a text file called images?
Does this mean I should block Scooter from certain directories in .htaccess? Let it chew 403's?
Or is this robots.txt-ignoring bot not really Scooter at all...?