| robots.txt expert question
Is it possible for robots.txt to disallow all but one page? |
DanielDiGriz
msg:1526021 | 7:58 pm on Aug 16, 2003 (gmt 0) |
In robots.txt I have: User-agent: *
Disallow: / What I want, instead, is to disallow all but one page, *without having to specifically disallow each page on the site* (pages are added constantly). Am I correct that robots.txt provides no way to do this, since the protocol is to disallow rather than allow?
|
jatar_k
msg:1526022 | 9:02 pm on Aug 16, 2003 (gmt 0) |
There is no allow, an option might be to disallow everything but that one page by doing them all explicitly. User-agent: *
Disallow: /index.html
Disallow: /pages/
Disallow: /img/
Disallow: /otherpage.html then have the page you want to allow in the root and disallow everything but.
|
jdMorgan
msg:1526023 | 9:34 pm on Aug 16, 2003 (gmt 0) |
DanielDiGriz, Welcome to WebmasterWorld [webmasterworld.com]! It can be helpful to split files which are allowed from those which are disallowed into different subdirectories. In this way, you can disallow a subdirectory path, and add content to it at will. Since the whole subdirectory is disallowed, you won't need to change robots.txt. Jim
|
DanielDiGriz
msg:1526024 | 11:38 pm on Aug 16, 2003 (gmt 0) |
Explicitly is not an option, but I like the subdirectory idea. Thank you for the responses. :)
|
|
|
