Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt and htaccess

         

agradation

5:26 pm on Jan 25, 2006 (gmt 0)

10+ Year Member



When I delete a page from my site, I'd like robots to remove it from their database, and humans to be redirected to a CGI which tells them what to do in that particular case.

If I add this line to .htaccess :
Redirect permanent /subdir/page.html (http+servername)/cgi-bin/404.cgi?some_ID

and this to robots.txt :
User-agent: *
Disallow: /cgi-bin

will I achieve my goal?
Thanks for help.

Dijkgraaf

3:40 am on Jan 26, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Some spiders won't recheck a URL received from a 301 redirect, so they may actually request that cgi-bin page.

agradation

10:23 pm on Jan 26, 2006 (gmt 0)

10+ Year Member



Thank you Dijkgraaf.

I am mostly worried about major spiders like Google or MSN. Will the majors act correctly?

My second worry is about the question mark in Redirect's target.

Dijkgraaf

11:17 pm on Jan 26, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I been thinking about setting up a series of pages to actually test the spiders behaviour on such issues as 301 redirects and others, I just haven't done it yet.
So I can't give a definite answer as yet.