I am creating a robots.txt file for a site which has an /images/ folder in the root directory which I want to Disallow with:
Disallow: /images/
But there are a number of folders in the root folder which have their own /images/ folders which I don't want to exclude, will the above line exclude them?
any assistance much appreciated.
jnmconsulting
3:29 am on Sep 30, 2004 (gmt 0)
If you use /images/ then the bot will not crawl the images folder in the root of the web only. to have it not crawl the image folder in the directory "test" the you would put
/test/images/
Hope that helps, do a search on robots.txt and you will find examples and plenty of info on how to use it.