Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Apache: execute "global" php script when robots.txt requested


lucy24 - 5:05 pm on Aug 20, 2012 (gmt 0)


Oh, I see the problem. You want to rewrite to a location outside the domain where the original request took place. Have a look at how mod_rewrite handles proxies (flags [P] and [PT]). In simplistic terms, that's an external redirect that is made to look like an internal rewrite. Stop screaming, g1, I did say "simplistic".

That's assuming you want the robot itself to be shipped off to some new location where you can Do Stuff on the fly. If all you need to do is get the information and process it for later use, that's a whole different question.


Thread source:: http://www.webmasterworld.com/robots_txt/4486406.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com