TheMadScientist - 7:16 pm on Mar 26, 2010 (gmt 0)
Let me expand a bit more on /directory/ from a server perspective, and why I can remove the slash and it's not a big deal, because there may be some confusion about what it is or does...
A request for /directory/ is a request for the default file of a directory. It is not a request for the 'index' of a directory specifially. Usually the default file of a directory is the 'index' of the directory, or an 'overview' of the directory, but (to the best of my knowledge) it is not required to be the index of the directory or an 'overview' of the directory and the location is not required to display any content whatsoever.
The following should be pre-pended with 'basically' or 'generally'.
In the server configuration file there is a setting for the default file the server should display when a root directory is requested. It is usually index.ext.
If there is no 'default' file in the directory, if directed to (most are) the server shows a 'generated' index of the files in the directory as a convenience. If the server is not directed to (or directed to not) create an index of the files in the directory it will generally display a '403 Forbidden' error page. (This is what happens on the w3.org site.)
The setting of the default file to show for directory requests is configurable and can be set to display *any* file in the directory by default for root directory requests. The file could be set as 'home.htm', 'foo.php', 'widget.pl, 'whateveryoufeellike.anyexthere' for the default file a root directory request displays to the user... It is usually index.ext, but it does not need to be.
Knowing the preceding:
If a person requests the root file of a directory on one of my sites they receive a 301 Redirect, because I have moved the default file (contents) to display for the directory request to one level above the actual directory in my site hierarchy on a page with the same name as the directory.
There is absolutely nothing wrong with doing this.
A request for /directory/ is a request for the server set default file of the directory, which is not required to be an index of the directory, even though it usually is. I have changed the location of the information usually presented for a root directory request (the default page to display) to a page one level above the directory with the name of the directory.
There is absolutely no reason why I cannot, as a visitor convenience, redirect the visitor to the new location of the information the default file for the directory would display if it was in the directory rather than serving a 403 Forbidden Error to the visitor when they request the default file for the directory.
Here's another way to look at it:
Most servers are configured to show a default file for a directory request, serve a generated index of the directory, or serve a 403 Forbidden Error for directory requests. I have configured mine to have the default file to show for a directory request be one level above the directory with the same page name as the directory.
There is no protocol or specification in the HTTP Docs I have ever seen saying I cannot change the location of the default file to display for directory requests to any location I feel like. (If you know of one, please provide a link to the resource.)
There are ways to help user agents (including search bots) understand where the index of a directory (or multiple directories) is located.
There is no 'breach of' or 'added confusion within' my directory structure or hierarchy by displaying an overview (index) of a directory one level above the directory in the site architecture.
It actually makes quite a bit of sense to me to have a page overviewing the directory one level above the directory itself...
Look at this:
The 'overview' of the sub-directory is located at:
The 'overview' of the directory is located at:
There may only be one accessible page in the directory above the directory a specific page is located in, but that's where you'll find the 'overviews' of the directories on my sites...
http://www.example.com/directory overviews what is inside http://www.example.com/directory/ : http://www.example.com/directory/sub-directory overviews what's inside http://www.example.com/directory/sub-directory/ there is no issue with what I am doing WRT HTTP Specification, and I personally have not ever had an issue with Search Engines even though I have been using this type of site organization for years.
If you choose to use the / on the end of your URLs, then cool, feel free to do so. It's completely up to you if you decide to show your all of your content as the 'default page' for a directory rather than on a specific page of your site, but redirecting one level up from the directory for the default page (EG /directory) is essentially the same as redirecting to the actual page name of the default file to be shown for directory requests (usually index.ext EG /directory/index.ext). Neither is wrong to do, but IMO one is definitely more attractive in the address bar, shorter and much easier for type ins...
Anyway, the reality is most sites I've seen with indexing issues have much more wrong with them than not using the /directory/ convention as an accessible URL, so my answer to the question the OP has (why not close the URL off with something?) is: Because there is no need and I don't like the trailing / on the URL, but each to their own on this one.
Just make sure you have good logical hierarchy in your URLs, solid link architecture, use link relationship markup within your documents and links if you think there could be confusion with some user agents (bots) or want some 'added assurances' and do what you think is best and most accessible for visitors, because all mechanisms are essentially the same, but IMO some look better and are easier to use than others.