Welcome to WebmasterWorld Guest from 126.96.36.199
Forum Moderators: goodroi
and I used a trailing slash. I saw here that apparently some browsers misinterpret that? Removing the slash somewhat bothers me, because what's stopping some *other* stupid robot from deciding that "foo/images" is a text file called images?
Does this mean I should block Scooter from certain directories in .htaccess? Let it chew 403's?
Or is this robots.txt-ignoring bot not really Scooter at all...?
IPs: 188.8.131.52, 184.108.40.206
That way, if it requests it with or without the ending slash, it should be denied.
stupid robot from deciding that "foo/images" is a text file called images?
As far as robots.txt is concerned, it doesn't matter what images is. It's supposed to just compare the disallow string with the beginning of the url.
Scooter seems to be obeying on my site.