Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: goodroi

Message Too Old, No Replies

multiple disallows

excluding multiple root directories.

     
1:34 pm on Oct 8, 2007 (gmt 0)

Junior Member

5+ Year Member

joined:Dec 22, 2006
posts:77
votes: 0


I'm just trying to make certain I have done this correctly.
It is not clear from what I have read if I can do multiple directory exclusions.

User-Agent: some bot
Disallow: /one/
Disallow: /two/
Disallow: /tree/
Disallow: /for/
Disallow: /five/

These are not tree directorys.
The all reside in the root directory and have their own structures.

ie:

Public HTML contains all 5 directories.

1:39 pm on Oct 10, 2007 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


hi michel,

i apologize but my caffeine is not working yet & my brain is still a little slow. are you trying to block files like:
http://www.example.com/one/
http://www.example.com/one/index.html
http://www.example.com/one/abc.html

or

http://www.example.com/level1/one/
http://www.example.com/level1/level2/one/abc.html

2:05 pm on Oct 10, 2007 (gmt 0)

New User

5+ Year Member

joined:Oct 10, 2007
posts:1
votes: 0


Yes. That's how you exclude multiple directories.
2:11 pm on Oct 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 13, 2003
posts:1281
votes: 0


Yes, this will work. To block directories within the subdirectories, you write it like this:

User-Agent: some bot
Disallow: /one/1.2/1.3/
Disallow: /two/
Disallow: /tree/
Disallow: /for/
Disallow: /five/scripts/
Disallow: /five/images/
Disallow: /five/includes/

Disallowing five like that will still leave the root of five and all other subdirectories open for spidering. Disallowing 1.3 like that only blocks that directory and leaves everything else in "one" open.

10:05 pm on Oct 10, 2007 (gmt 0)

Junior Member

5+ Year Member

joined:Dec 22, 2006
posts:77
votes: 0


You answered my question perfectly.
I'm sitting behind a geoip block of the site.

The last time I did not limit 'the bigest american search engines" access. I got torn apart by unwanted traffic the last time and it cost me a fortune.

So what I have done is expose the index page without the geoip block. Therfore allowing all search engines to access it.

But keeping the rest of the site in other directories with the geiop block in place.

This way only the first page shows up on search engines. And my comment that they are out of my service area will only need to be displayed on the front page.