Forum Moderators: goodroi

Message Too Old, No Replies

Bot cn exclude this directory?

         

tintin74

8:19 am on Aug 22, 2005 (gmt 0)

10+ Year Member



Hello,

I've recently include this line in my robotx.txt:

Disallow: /example

Because I need to exclude some files like example.htm, example.php ...

Unfortunatly these file are store in the directory /example_cars/

Do you think , the bot will not spider the entire directory or only the files named example.php, example.htm...

Dijkgraaf

10:42 pm on Aug 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The bot won't spider anything starting with /example, so by that rule anything in /example_cars/ will also be excluded.
However if you changed it to
Disallow: /example_cars/example.
(note the full stop at the end), then it won't exclude everthing in that directory but would exclude example.php, example.htm or example.html etc. in that directory.