homepage Welcome to WebmasterWorld Guest from 54.196.196.62
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
replacing robots.txt with a CGI
theperlyking




msg:1529593
 9:21 am on Apr 25, 2001 (gmt 0)

I would like to redirect robots.txt to a cgi so that I can log activity better (I dont have access to server logs :( ) and potentially have the option to do fancier things.

Does anyone know if there will be any pitfalls to this? Is there anyone doing this?

Thanks.

 

littleman




msg:1529594
 4:58 pm on Apr 25, 2001 (gmt 0)

It really depends on how you do it. Do you have mod_rewrite? I don't think you want to use a redirect header.

theperlyking




msg:1529595
 5:18 pm on Apr 25, 2001 (gmt 0)

Yep I have mod_rewrite, it's what i'm planning to use.
I can't imagine the SE balking at it but I was just wondering if anyone knew better.

littleman




msg:1529596
 5:22 pm on Apr 25, 2001 (gmt 0)

Then this would do the trick.
RewriteEngine on
RewriteBase /
RewriteRule ^robots\.txt$ robots.pl

I do something similar.

theperlyking




msg:1529597
 6:05 pm on Apr 25, 2001 (gmt 0)

Good, I know how to do it I was just scared :)

Thanks

littleman




msg:1529598
 6:17 pm on Apr 25, 2001 (gmt 0)

Well, I guess you would judging from your name ;).

If you are real paranoid, I'd take the time to make sure the script's header matches up. It might be wise to make the script a 'nph-' and craft the header info your self depending on what the scripts output is.

theperlyking




msg:1529599
 7:42 pm on Apr 25, 2001 (gmt 0)

Well my name is just fun, I kludge rather than code :)

Thanks for the real world advice I'll have a go.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved