Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Need to redirect non-spiders with htaccess



6:11 am on Oct 21, 2003 (gmt 0)

10+ Year Member

I have a script that creates a text only version of my Perl based forum for robots to spider.

I am currently using a javascript redirect with a meta refresh in the <no script> tag to send visitors to the real board.

I know Google and maybe others do not like pages that redirect, so I want to use ModRewrite to redirect everyone except top bots to the real forum.

While I have a basic knowledge of ModRewrite, I'm not up to this. Any help is appreciated.


3:00 pm on Oct 21, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Generally, we prefer to help people DEBUG their mod_rewrite code at WebmasterWorld, not write it for them. Although someone may jump in here and offer a solution, you could start with this Intro to mod_rewrite [webmasterworld.com] (and follow the links it includes if it's too basic) or try a post in the Commercial Exchange forum.


3:14 pm on Oct 21, 2003 (gmt 0)

10+ Year Member

hi guillermo5000.

i see that you write perl scripts. i loaded shed loads of spider c-blocks into my database then wrote a script that checked if the visitor was a spider or not. depending on this check you can serve different content.

you can maintain a table of visitorips which you can purge say every half an hour.

i actually used this for logging purposes rather than serving different content but the principle is the same. i now have a system where i have visits and pageviews for both humans and spiders.

why not implement something like this in perl? just remember to update your spider list every now and then.


5:31 pm on Oct 21, 2003 (gmt 0)

10+ Year Member

incy: Thanks, but I do not write perl. I just use a 3rd party board that is written in Perl.

Dave: Thank you. I'll look elsewhere.


Featured Threads

Hot Threads This Week

Hot Threads This Month