Forum Moderators: goodroi

Message Too Old, No Replies

disallow every 'filename' except.

disallow every 'filename' except...

         

decbrad

11:08 am on Aug 1, 2006 (gmt 0)

10+ Year Member



Hi there ...can somebody advise me on this!

I am trying to write a robots.txt file that will disallow the SE's to index my website with the exception of two pages, the index.asp and the about.asp pages. Is this possible? ...here's what I have at the moment!

User-agent: *
Disallow: /includes/
Disallow: /db/
Disallow: /images/

...it would be great if I could disallow everything 'except' certain filenames.

Just out of curiousity is this approach advisable or from a ranking point of view should I allow access to the full site. I would rather concentrate on optimising the 2 main 'sales' pages for the SE's

thanks for any assistance...

rgds,
Decbrad

cheesehead2

4:12 am on Aug 4, 2006 (gmt 0)

10+ Year Member



I'm not sure how do this for other SE, but for Google you could do this:

"Googlebot recognizes an extension to the robots.txt standard called Allow."
[google.com...]


User-Agent: Googlebot
Disallow: /
Allow: /index.asp
Allow: /about.asp

You could aslo organize your site where all the internal pages are under a subdirectory, but I think allowing only two pages is a bad idea.