homepage Welcome to WebmasterWorld Guest from 54.161.214.221
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque

Webmaster General Forum

    
Session init redirection and its impacts on search engine bots
Advice needed
dnimrodx




msg:3331773
 7:08 pm on May 5, 2007 (gmt 0)

Hello,

I have a website that at every session start redirects to a page that checks to see the client browser supports session cookies and has javascript enabled. After these checks are performed this page is never invoked again.

What I would like to know is what the impacts are of such a redirection at session-start on web crawling bots? Can they handle it well, or should I just abandon this 'session-check' altogether and adopt a more conventional way (no checking)?

Thank you.

 

dataguy




msg:3332559
 12:54 am on May 7, 2007 (gmt 0)

Even thought the gbot has been reported to be able to read cookies, I don't think there are any crawlers that will allow the session to be saved, so chances are every crawler that hits a page of your site will be redirected, every time.

Doesn't sound like a good idea to me.

jdMorgan




msg:3332570
 1:08 am on May 7, 2007 (gmt 0)

Right. Never give a search engine spider a session unless you want big trouble.

Jim

dnimrodx




msg:3332825
 10:48 am on May 7, 2007 (gmt 0)

Never give a search engine spider a session unless you want big trouble.

How exactly can I find out whether a visitor to my website is a spider bot? Mind you, I've submitted my website to a number of different search engines and not just Google or Yahoo...

Thanks to all of your replies.

Dave

jdMorgan




msg:3332865
 12:19 pm on May 7, 2007 (gmt 0)

Simple method: Examine the User-agent header sent by the client in its HTTP requests to your server.
Better: Examine that 'claimed' User-agent, and verify that with a reverse-DNS lookup on the client's IP address to be sure the User-agent is a genuine one and not a spoof.

How you do this depends on what server you're running on and/or what language you're using to write your code.

If you're using an off-the-shelf software package, have a look through the documentation and/or the on-line FAQs. This is a common situation, and pre-made solutions may be readily available.

Jim

dnimrodx




msg:3332868
 12:24 pm on May 7, 2007 (gmt 0)

Simple method: Examine the User-agent header sent by the client in its HTTP requests to your server.
Better: Examine that 'claimed' User-agent, and verify that with a reverse-DNS lookup on the client's IP address to be sure the User-agent is a genuine one and not a spoof.

How you do this depends on what server you're running on and/or what language you're using to write your code.

If you're using an off-the-shelf software package, have a look through the documentation and/or the on-line FAQs. This is a common situation, and pre-made solutions may be readily available.

Jim

I'm using custom made scripts in PHP and using Apache v2 on Linux, so I don't think it will be a problem to implement this. Thanks for the help Jim.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved