Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

How would I disallow Google...

...and allow all others?



11:44 pm on Nov 27, 2004 (gmt 0)

10+ Year Member

I have some duplicate content that I don't want Google (to keep) penalizing me for, and I was wondering how to accomplish the following with a robots.txt:

1) Disallow Google totally.

2) Allow Googlebot into index page (and links pages), but disallow to other sub-pages.



11:53 pm on Nov 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member


User-agent: googlebot
Disallow: /

for the second u may use meta tags


5:35 pm on Nov 28, 2004 (gmt 0)

10+ Year Member

Is there a way to disallow subpages with a robots.txt? Changing individual pagess can be a bit tedious.



Featured Threads

Hot Threads This Week

Hot Threads This Month