Forum Moderators: open

Message Too Old, No Replies

Can my robots.txt fitler out individual files?

         

irock

6:26 am on Feb 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi I would like to disable all spiders from grabbnig my stuff inside these files (/forums/printthread.php, /forums/member.php).

User-agent: *
Disallow: /forums/printthread.php
Disallow: /forums/member.php

Here's my robots.txt. Is this in the correct format?

Thanks VERY MUCH for your help!

jdMorgan

6:36 am on Feb 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



irock,

disable all spiders

Your posted disallows are correct and will keep out those spiders which obey the robots exclusion standard. However, this won't keep out robots which choose to ignore robots.txt.

Once you upload the file, use this robots.txt checker [searchengineworld.com] to be sure it is correct.

Jim

irock

9:34 am on Feb 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Is Google one of those spiders that obey robot rules?

andreasfriedrich

12:52 pm on Feb 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes.