Forum Moderators: mack

Message Too Old, No Replies

Can Robots.txt have an include command?

Question on whether Robots.txt have an include command?

         

franjavier

3:57 pm on Jan 16, 2004 (gmt 0)

10+ Year Member



My robots.txt it as follows:

------------------------
User-agent: *
Disallow: /folder1/web/
------------------------

In folder /folder1/web/ I have many files that i want to exclude with the exception of one file inc.html (which it would be too difficult to change of folder)

My question is does Robots text support and Disallow/allow file option?

That is, can it work something like this:

------------------------
User-agent: *
Disallow: /folder1/web/
Allow: /folder1/web/inc.html
------------------------

pmac

8:02 pm on Jan 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi franjavier, and welcome to WebmasterWorld.

No, there is no allow option that I am aware of. This is a real good robots txt tutorial [searchengineworld.com]