When I delete a page from my site, I'd like robots to remove it from their database, and humans to be redirected to a CGI which tells them what to do in that particular case.
If I add this line to .htaccess : Redirect permanent /subdir/page.html (http+servername)/cgi-bin/404.cgi?some_ID
and this to robots.txt : User-agent: * Disallow: /cgi-bin
will I achieve my goal? Thanks for help.
Dijkgraaf
3:40 am on Jan 26, 2006 (gmt 0)
Some spiders won't recheck a URL received from a 301 redirect, so they may actually request that cgi-bin page.
agradation
10:23 pm on Jan 26, 2006 (gmt 0)
Thank you Dijkgraaf.
I am mostly worried about major spiders like Google or MSN. Will the majors act correctly?
My second worry is about the question mark in Redirect's target.
Dijkgraaf
11:17 pm on Jan 26, 2006 (gmt 0)
I been thinking about setting up a series of pages to actually test the spiders behaviour on such issues as 301 redirects and others, I just haven't done it yet. So I can't give a definite answer as yet.