Forum Moderators: goodroi
I am trying to write a robots.txt file that will disallow the SE's to index my website with the exception of two pages, the index.asp and the about.asp pages. Is this possible? ...here's what I have at the moment!
User-agent: *
Disallow: /includes/
Disallow: /db/
Disallow: /images/
...it would be great if I could disallow everything 'except' certain filenames.
Just out of curiousity is this approach advisable or from a ranking point of view should I allow access to the full site. I would rather concentrate on optimising the 2 main 'sales' pages for the SE's
thanks for any assistance...
rgds,
Decbrad
"Googlebot recognizes an extension to the robots.txt standard called Allow."
[google.com...]
User-Agent: Googlebot
Disallow: /
Allow: /index.asp
Allow: /about.asp
You could aslo organize your site where all the internal pages are under a subdirectory, but I think allowing only two pages is a bad idea.