homepage Welcome to WebmasterWorld Guest from 54.167.138.53
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How would I disallow Google...
...and allow all others?
Rollo

10+ Year Member



 
Msg#: 501 posted 11:44 pm on Nov 27, 2004 (gmt 0)

I have some duplicate content that I don't want Google (to keep) penalizing me for, and I was wondering how to accomplish the following with a robots.txt:

1) Disallow Google totally.

2) Allow Googlebot into index page (and links pages), but disallow to other sub-pages.

Thanks!

 

benevolent001

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 501 posted 11:53 pm on Nov 27, 2004 (gmt 0)

totally

User-agent: googlebot
Disallow: /

for the second u may use meta tags

Rollo

10+ Year Member



 
Msg#: 501 posted 5:35 pm on Nov 28, 2004 (gmt 0)

Is there a way to disallow subpages with a robots.txt? Changing individual pagess can be a bit tedious.

Thanks!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved