Forum Moderators: open

Message Too Old, No Replies

Meta tag stop code to prevent SEspidering of some pages?

Find out if a stop code exists to limit SE spidering of some website pages.

         

brmarketing

9:12 pm on Nov 20, 2002 (gmt 0)

10+ Year Member



Is there a meta tag stop code I can use to limit SE spidering of our password-protected client access website pages while still allowing for spidering of the overall site?

jatar_k

9:17 pm on Nov 20, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



Welcome to WebmasterWorld [webmasterworld.com] brmarketing

have you looked at having a robots.txt [searchengineworld.com] file on your site to limit where spiders access?

<added>If it is password protected how are spiders getting by the login script?

Romeo

9:19 pm on Nov 20, 2002 (gmt 0)

10+ Year Member



You can try to put a
<meta name="robots" content="noindex,nofollow">
into the head section of your page, but it depends on the implementation of the SE if it will respect this. AFAIK, Google does ...

Regards,
R.

brmarketing

2:54 pm on Nov 21, 2002 (gmt 0)

10+ Year Member



Thanks! We'll investigate using a robots.txt file or a robots meta tag.

pageoneresults

5:20 pm on Nov 21, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



> You can try to put a
<meta name="robots" content="noindex,nofollow">

Romeo is correct. You can also use the shorter version...

<meta name="robots" content="none">

Which is equivalent to noindex, nofollow.

Romeo, Romeo, where art thou? Welcome! I'm surprised that username was available. Welcome to brmarketing too!