homepage Welcome to WebmasterWorld Guest from 54.166.113.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Will the following robots.txt disallow all robots BUT Google?
georgec

10+ Year Member



 
Msg#: 357 posted 10:33 am on Apr 7, 2004 (gmt 0)

Just wondering, will the following robots.txt correctly disallow all robots but Google:

User-agent: *
Disallow: /
User-agent: GoogleBot
Disallow:

Logically I think this is correct, though on Google's FAQ, they seem to want us to use "allow" instead of "Disallow:"

Thanks for any insight.

 

SEOMike

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 357 posted 2:11 pm on Apr 8, 2004 (gmt 0)

From the way I understand it, Googlebot needs to see it's rule first. I usually code mine around the following example:

# Keeps Googlebot out of targeted pages & private stuff
#
User-agent: Googlebot
Disallow: /content/
Disallow: /_private/
Disallow: /includes/
Disallow: /scripts/
#
# Keeps the rest of the engines out of private stuff
#
User-agent: *
Disallow: /_private/
Disallow: /includes/
Disallow: /scripts/

Hope this helps.

sidyadav

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 357 posted 1:15 pm on Apr 9, 2004 (gmt 0)

Yeah, when Googlebot visits your robots.txt file, it needs to know first whether its allowed or not. So if you enter the "dis-allow all robots code" first, it will simple read just that and not whats after it. But if it knows that its allowed, it will read and follow that, forgetting about the next line of code which dis-allows all robots.

Here's the valid code which dis-allows others while allowing only Googlebot:
User-agent: googlebot
Disallow:

User-agent: *
Disallow: /

Alternatively, you can use this one [searchengineworld.com] which will allow only the spiders which are "nice".

Sid

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved