homepage Welcome to WebmasterWorld Guest from 54.227.160.102
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
replacing robots.txt with a CGI
theperlyking

10+ Year Member



 
Msg#: 119 posted 9:21 am on Apr 25, 2001 (gmt 0)

I would like to redirect robots.txt to a cgi so that I can log activity better (I dont have access to server logs :( ) and potentially have the option to do fancier things.

Does anyone know if there will be any pitfalls to this? Is there anyone doing this?

Thanks.

 

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 119 posted 4:58 pm on Apr 25, 2001 (gmt 0)

It really depends on how you do it. Do you have mod_rewrite? I don't think you want to use a redirect header.

theperlyking

10+ Year Member



 
Msg#: 119 posted 5:18 pm on Apr 25, 2001 (gmt 0)

Yep I have mod_rewrite, it's what i'm planning to use.
I can't imagine the SE balking at it but I was just wondering if anyone knew better.

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 119 posted 5:22 pm on Apr 25, 2001 (gmt 0)

Then this would do the trick.
RewriteEngine on
RewriteBase /
RewriteRule ^robots\.txt$ robots.pl

I do something similar.

theperlyking

10+ Year Member



 
Msg#: 119 posted 6:05 pm on Apr 25, 2001 (gmt 0)

Good, I know how to do it I was just scared :)

Thanks

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 119 posted 6:17 pm on Apr 25, 2001 (gmt 0)

Well, I guess you would judging from your name ;).

If you are real paranoid, I'd take the time to make sure the script's header matches up. It might be wise to make the script a 'nph-' and craft the header info your self depending on what the scripts output is.

theperlyking

10+ Year Member



 
Msg#: 119 posted 7:42 pm on Apr 25, 2001 (gmt 0)

Well my name is just fun, I kludge rather than code :)

Thanks for the real world advice I'll have a go.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved