homepage Welcome to WebmasterWorld Guest from 54.196.159.11
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
multiple disallows
excluding multiple root directories.
Michel Samuel




msg:3471776
 1:34 pm on Oct 8, 2007 (gmt 0)

I'm just trying to make certain I have done this correctly.
It is not clear from what I have read if I can do multiple directory exclusions.

User-Agent: some bot
Disallow: /one/
Disallow: /two/
Disallow: /tree/
Disallow: /for/
Disallow: /five/

These are not tree directorys.
The all reside in the root directory and have their own structures.

ie:

Public HTML contains all 5 directories.

 

goodroi




msg:3473735
 1:39 pm on Oct 10, 2007 (gmt 0)

hi michel,

i apologize but my caffeine is not working yet & my brain is still a little slow. are you trying to block files like:
http://www.example.com/one/
http://www.example.com/one/index.html
http://www.example.com/one/abc.html

or

http://www.example.com/level1/one/
http://www.example.com/level1/level2/one/abc.html

SEODoctor




msg:3473757
 2:05 pm on Oct 10, 2007 (gmt 0)

Yes. That's how you exclude multiple directories.

SEOMike




msg:3473764
 2:11 pm on Oct 10, 2007 (gmt 0)

Yes, this will work. To block directories within the subdirectories, you write it like this:

User-Agent: some bot
Disallow: /one/1.2/1.3/
Disallow: /two/
Disallow: /tree/
Disallow: /for/
Disallow: /five/scripts/
Disallow: /five/images/
Disallow: /five/includes/

Disallowing five like that will still leave the root of five and all other subdirectories open for spidering. Disallowing 1.3 like that only blocks that directory and leaves everything else in "one" open.

Michel Samuel




msg:3474247
 10:05 pm on Oct 10, 2007 (gmt 0)

You answered my question perfectly.
I'm sitting behind a geoip block of the site.

The last time I did not limit 'the bigest american search engines" access. I got torn apart by unwanted traffic the last time and it cost me a fortune.

So what I have done is expose the index page without the geoip block. Therfore allowing all search engines to access it.

But keeping the rest of the site in other directories with the geiop block in place.

This way only the first page shows up on search engines. And my comment that they are out of my service area will only need to be displayed on the front page.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved