Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
All of the promotionional pages explaining my services are in a folder called website_design. My clients websites each have their own folder e.g. left_handed_pencils. I don't want spiders that visit my url to index my clients folders (my clients all have their own url).
I know about using 'Disallow' - is it possible to use 'Allow' or something with that meaning so that I don't have to keep changing the robots.txt file?
clients/nasa well not yet :)
It would be easier if I could just use some kind of 'Allow only this folder' instruction in my robots.txt