Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Apache: execute "global" php script when robots.txt requested
g1smd - 8:19 am on Aug 20, 2012 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/4486406.htm
I posted a complete robots.txt file logger script here only a few days ago. It makes a new file once per week.
It works via a RewriteRule. It also detects requested hostname and could be easily configured to generate a separate file per site.
Brought to you by WebmasterWorld: http://www.webmasterworld.com