homepage Welcome to WebmasterWorld Guest from 54.196.62.132
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Blocking Google Only
From certain sub domains
Moncao




msg:3307444
 1:54 pm on Apr 10, 2007 (gmt 0)

Being a victim of Google's new algo, I need to stop them (Google) from crawling certain sub directories of my site. How do I do this?

 

goodroi




msg:3308352
 1:14 pm on Apr 11, 2007 (gmt 0)

Add this to your robots.txt
User-agent: Googlebot
Disallow: /subdirectory1/
Disallow: /subdirectory2/
Disallow: /subdirectoryetc/

[google.com...]

To verify your robots.txt is correct you can use Google Webmaster Central.

Moncao




msg:3310046
 7:06 am on Apr 13, 2007 (gmt 0)

Hi Goodroi

Thanks, but Google's own pages say to leave sub-directories /open without the traling backslash, no /closed/ as with everyone else.

goodroi




msg:3310599
 7:18 pm on Apr 13, 2007 (gmt 0)

to further complicate things you can look at their own robots.txt file (http://www.google.com/robots.txt) which uses both styles :)

encyclo




msg:3310609
 7:25 pm on Apr 13, 2007 (gmt 0)

I believe there is a difference between the two styles (please correct me if I'm wrong!). As I understand it:

Disallow: /foo

Matches /foo/ and /foo.html

Whereas:

Disallow: /foo[b]/[/b]

Matches the directory only.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved