Disallow all Directories

[Omega1Megs]

I would like to know if I can exclude all directories with a single line and only allow the html files to be allowed. Instead of listing each directory out individually.

[jdMorgan]

In the Standard for Robot Exclusion, there is no "Allow" directive. Some robots have defined "extensions" to the Standard that they support, but only a few robots support these extensions. So it's a mess.

If you site is small and/or new, then consider re-arranging your directory structure so that robots can be controlled by directory, not by file-type. If it's a large established site, and cahanges to the existing directory structure are not feasible, then you'll just have to list out the Disallowed files one-by-one if they are in the same directory as "allowed" content.

Jim