Forum Moderators: phranque
[example.com...]
to
[example.com...]
Do spiders still view the directory as /cgi-bin/ or do they now view it as /catalog/? In other words, if I disallow spiders from /cgi-bin/ will they still follow the /catalog/ URLs?
BTW, I put this in the robots.txt forum, but am not sure anyone who answered knew what I was talking about, so I am putting it here.
Unless you use redirection (R flag), mod_rewrite is completely internal to the server
I'm sorry to sound dumb, but I am still getting this rewrite stuff down. Could you please explain that a little bit more?
No
>> or do they now view it as /catalog/?
Yes
>> In other words, if I disallow spiders from /cgi-bin/ will they still follow the /catalog/ URLs?
Yes. Unless you disallow catalog, of course. What rogerdp said was that if you have made a 301 or 302 redirect, then it would not work, and he's probably right about that. So, if there's no [R] in your RewriteRule then there's no problem.
Heres's an example of what it should not look like:
RewriteRule something/ http.//something.else/ [R=301]
This is what you want in stead:
RewriteRule something/ something.else/
...without the [R] part at the end, that is. Also without the "http.//" as it is an internal rewrite, ie. a rewrite between two files on the same domain (i've used a dot in stead of a colon so that it would not become a link in the post). Hope this makes some sense, otherwise just ask ;)
