slurp ignoring disallow?

-or have i got it wrong?

[tallis]

i have a link to make a comment on every page, which i'd prefer not be indexed, and i've been using this:

User-agent: *
Disallow:
Disallow: comment.php

but i've noticed that slurp is still visiting those pages, have i got it wrong, or is there some other way to prevent those pages from being indexed?

they're mostly in different directories from each other, so i'm not sure if i'm supposed to be making that clear somehow, any ideas?

tallis

[Clint]

I think you must put the slash / before the path, and you have disallow twice. I think it should be like this:

User-agent: *
Disallow: /comment.php

[tallis]

oops, that second disallow is a pasting error. ;-/

thanks for that, do you know if having a slash in front of the filename stop it from being spidered no matter how deep it is in the directories? -or do i need to specify somehow that it's several deep?

tallis

[Clint]

In that example, that will stop the bots from spidering ONLY root/comment.php. It's just like any other path, if that page is located elsewhere besides yourdomain.com/comment.php and you don't want it spidered, then you have to add each path.

User-agent: *
Disallow: /comment.php
Disallow: /whatever/comment.php
Disallow: /whatever/whatever/comment.php
Disallow: /whatever2/comment.php

etc.