homepage Welcome to WebmasterWorld Guest from 54.196.197.153
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Will the following robots.txt disallow all robots BUT Google?
georgec




msg:1529001
 10:33 am on Apr 7, 2004 (gmt 0)

Just wondering, will the following robots.txt correctly disallow all robots but Google:

User-agent: *
Disallow: /
User-agent: GoogleBot
Disallow:

Logically I think this is correct, though on Google's FAQ, they seem to want us to use "allow" instead of "Disallow:"

Thanks for any insight.

 

SEOMike




msg:1529002
 2:11 pm on Apr 8, 2004 (gmt 0)

From the way I understand it, Googlebot needs to see it's rule first. I usually code mine around the following example:

# Keeps Googlebot out of targeted pages & private stuff
#
User-agent: Googlebot
Disallow: /content/
Disallow: /_private/
Disallow: /includes/
Disallow: /scripts/
#
# Keeps the rest of the engines out of private stuff
#
User-agent: *
Disallow: /_private/
Disallow: /includes/
Disallow: /scripts/

Hope this helps.

sidyadav




msg:1529003
 1:15 pm on Apr 9, 2004 (gmt 0)

Yeah, when Googlebot visits your robots.txt file, it needs to know first whether its allowed or not. So if you enter the "dis-allow all robots code" first, it will simple read just that and not whats after it. But if it knows that its allowed, it will read and follow that, forgetting about the next line of code which dis-allows all robots.

Here's the valid code which dis-allows others while allowing only Googlebot:
User-agent: googlebot
Disallow:

User-agent: *
Disallow: /

Alternatively, you can use this one [searchengineworld.com] which will allow only the spiders which are "nice".

Sid

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved