directory and subdirectory access

allowing access to subdirectory but not main directory

[Mr Cool Dude]

Hello

Suppose I had a directoy and subdirectory like this:

www.randomwebsite.com/media/
www.randomwebsite.com/media/images/

If I want to block robots from accessing the directory /media/ but I DO want them to index the content in the subdirectory /media/images/ how do I set up the robots.txt file?

I tried

Disallow: /media

and it blocked everything including the /media/images/ subdirectory.

Then I tried

Disallow: /media/

And it still prevented the /media/images/ subdirectory from being indexed. And help would be appreciated. Thanks.

[goodroi]

Disallow /media/ and allow /media/images/. That should make the search engines do what you want.

[jdMorgan]

But do that only within a record addressed to search engines that support the non-standard "Allow" directive... Many robots don't support it.

Alternatively, use the server configuration to deny access to the /media directory index. For example, on Apache, you could put

 Options -Indexes 

in your .htaccess file to accomplish this.

Jim