Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
multiple disallows excluding multiple root directories. Michel Samuel msg:3471776 1:34 pm on Oct 8, 2007 (gmt 0) I'm just trying to make certain I have done this correctly. It is not clear from what I have read if I can do multiple directory exclusions.
User-Agent: some bot
Disallow: /one/ Disallow: /two/ Disallow: /tree/ Disallow: /for/ Disallow: /five/
These are not tree directorys.
The all reside in the root directory and have their own structures.
Public HTML contains all 5 directories.
goodroi msg:3473735 1:39 pm on Oct 10, 2007 (gmt 0)
i apologize but my caffeine is not working yet & my brain is still a little slow. are you trying to block files like:
http://www.example.com/one/ http://www.example.com/one/index.html http://www.example.com/one/abc.html
http://www.example.com/level1/level2/one/abc.html SEODoctor msg:3473757 2:05 pm on Oct 10, 2007 (gmt 0)
Yes. That's how you exclude multiple directories. SEOMike msg:3473764 2:11 pm on Oct 10, 2007 (gmt 0)
Yes, this will work. To block directories within the subdirectories, you write it like this:
User-Agent: some bot
Disallow: /one/1.2/1.3/ Disallow: /two/ Disallow: /tree/ Disallow: /for/ Disallow: /five/scripts/ Disallow: /five/images/ Disallow: /five/includes/
Disallowing five like that will still leave the root of five and all other subdirectories open for spidering. Disallowing 1.3 like that only blocks that directory and leaves everything else in "one" open.
Michel Samuel msg:3474247 10:05 pm on Oct 10, 2007 (gmt 0)
You answered my question perfectly. I'm sitting behind a geoip block of the site.
The last time I did not limit 'the bigest american search engines" access. I got torn apart by unwanted traffic the last time and it cost me a fortune.
So what I have done is expose the index page without the geoip block. Therfore allowing all search engines to access it.
But keeping the rest of the site in other directories with the geiop block in place.
This way only the first page shows up on search engines. And my comment that they are out of my service area will only need to be displayed on the front page.