Welcome to WebmasterWorld Guest from 107.21.175.43

Forum Moderators: goodroi

Message Too Old, No Replies

multiple disallows

excluding multiple root directories.

     

Michel Samuel

1:34 pm on Oct 8, 2007 (gmt 0)

5+ Year Member



I'm just trying to make certain I have done this correctly.
It is not clear from what I have read if I can do multiple directory exclusions.

User-Agent: some bot
Disallow: /one/
Disallow: /two/
Disallow: /tree/
Disallow: /for/
Disallow: /five/

These are not tree directorys.
The all reside in the root directory and have their own structures.

ie:

Public HTML contains all 5 directories.

goodroi

1:39 pm on Oct 10, 2007 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



hi michel,

i apologize but my caffeine is not working yet & my brain is still a little slow. are you trying to block files like:
http://www.example.com/one/
http://www.example.com/one/index.html
http://www.example.com/one/abc.html

or

http://www.example.com/level1/one/
http://www.example.com/level1/level2/one/abc.html

SEODoctor

2:05 pm on Oct 10, 2007 (gmt 0)

5+ Year Member



Yes. That's how you exclude multiple directories.

SEOMike

2:11 pm on Oct 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, this will work. To block directories within the subdirectories, you write it like this:

User-Agent: some bot
Disallow: /one/1.2/1.3/
Disallow: /two/
Disallow: /tree/
Disallow: /for/
Disallow: /five/scripts/
Disallow: /five/images/
Disallow: /five/includes/

Disallowing five like that will still leave the root of five and all other subdirectories open for spidering. Disallowing 1.3 like that only blocks that directory and leaves everything else in "one" open.

Michel Samuel

10:05 pm on Oct 10, 2007 (gmt 0)

5+ Year Member



You answered my question perfectly.
I'm sitting behind a geoip block of the site.

The last time I did not limit 'the bigest american search engines" access. I got torn apart by unwanted traffic the last time and it cost me a fortune.

So what I have done is expose the index page without the geoip block. Therfore allowing all search engines to access it.

But keeping the rest of the site in other directories with the geiop block in place.

This way only the first page shows up on search engines. And my comment that they are out of my service area will only need to be displayed on the front page.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month