lucy24 - 10:42 am on Sep 29, 2012 (gmt 0)

To a human:
A redirect means that your browser's address bar changes, and you are now on a different page than the one you originally asked for. (Browsers ordinarily do this without asking your permission. The site says "Go around the back" and your browser obliges.)

A rewrite means that your address bar doesn't change, but you're seeing content that lives somewhere else. A special kind of rewrite that everyone has met is the 404 page: Your address bar will say www.example.com/ directory/ pagename.html or whatever you typed in, but the page you are looking at will be an error page that lives somewhere else entirely.

To a robot:
A redirect is a message that the stuff you want to see is somewhere else. Robots, unlike humans, can choose not to follow redirects. That is: they can't ignore the redirect and barge on to the page they originally asked for. But they can go away and try the new URL later-- or not at all. ("Oh, right, /foobar.html. I was there yesterday.")

But robots are powerless against rewrites. They don't know they've been rewritten, any more than humans do, and they can't ignore the rewrite.

If you redirect a robot to robots.txt it will say (in Robot) "Haha, very funny, I'll come back later when you're in a better mood". If you rewrite it to robots.txt, it will end up there whether it wants to or not.

How To
In mod_rewrite, there are two overlapping ways to create a redirect. One is to include the full protocol-plus domain in the target: http://www.example.com et cetera. The other is to use a flag saying [R]. Or, preferably, [R=301]. Either one by itself will turn a rewrite into a redirect, but you should do both together, for reasons that have nothing to do with bing.

To make a rewrite, you simply leave out both of those things. Keep the [L] flag, because you always use it. But change the target to say only /robots.txt

And then sit back and wait for them to start yapping about Duplicate Content as they see that every one of your pages says the exact same thing.