Forum Moderators: goodroi

Message Too Old, No Replies

Can someone tell me how to ........

         

Lex_Luther

4:06 pm on May 3, 2005 (gmt 0)

10+ Year Member



disallow slurp from indexing or following, and allow google bot to index, but not to follow?

What would the robots command for this be? Actually I would prefer to ban all robots except google bot.

I search around the web and I cant find the exact robot command this would take, and I unforntunatly am not familiar enough on the subject to figure it out. write this.

Thanks

Span

8:41 pm on May 3, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



How do I block all crawlers except Googlebot from my site? [google.com]

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /

But if you want Gbot to index your site but not follow links you have to put a robots meta tag on every page:
<meta name="robots" content="index,nofollow" />
But then how is a bot supposed to index your site if it is not allowed to follow links?

ncw164x

9:00 pm on May 3, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This is what you need to disallow all other spiders and to give googlebot access to your site

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

Like Span says its pointless to have index,nofollow on your pages otherwise googlebot can't index your site