Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Can robots.txt block a directory but not its subfolder?

         

realmaverick

8:21 pm on Oct 25, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi guys,

I want to block Google from /directory/ but not /directory/subfolder/

I have:

Disallow /directory/

But I believe it's also blocking /directory/subfolder/

As the subfolder contains all of my images. About a million of them, and not a single one is indexed in Google images. Yet images hosted in other folders are all indexed just fine.

Also, Google previews never show images inside the /directory/subfolder/, just a place holder.

Any ideas how I can achieve the above?

Can I simply specify Allow /directory/subfolder/ after the Disallow statement?

I'm a little confused as to how it works.

Dijkgraaf

8:46 pm on Oct 25, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



A disallow says don't ask for anything starting with, so yes, the sub directory will be blocked also.
No you can't simply allow /directory/subfolder/
I would say move the subfolder.

realmaverick

8:51 pm on Oct 25, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hmm I thought there was a way to allow a subfolder. Moving the folder isn't really viable, not without a lot of work.

netmeg

8:57 pm on Oct 25, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Is there anything in the directory (not the subfolder) that's got a link to it?

You could maybe put a blank index.html in the directory, and put a NOINDEX robots meta tag on it, and then remove the directory from robots.txt?

(you could test it on one of your image URLs in GWT ahead of time to see if it works)

realmaverick

1:18 am on Oct 27, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes the exact issue is this:

I no indexed my entire /forum/ because it was causing too many errors in WMT. There is an images folder inside /forum/ that is used in my galleries.

There are way too many files that would need updating.

In a new version of my site, this issue is actually fixed. However, it's not going to be released for another month or so.

So is this basically impossible?

lucy24

4:12 am on Oct 27, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



You're moving in a month and then everything will work? And the affected directory contains only images? Fer hevvins sakes. Let them go un-indexed for another month.

Besides, it will probably turn out that half of your image-searching visitors are just looking for things to hotlink to.