Hello,
I wish to know if several robots.txt will work? I don't plan to disallow all the directories I don't want to be crawled from the root (/www/robots.txt) robots.txt because I don't want someone to load this file and discover the urls of some directories.
So I plan to put several robots.txt additionally to the one in the root (/www/directory/robots.txt ....) .
Will it work? Does crawler only look for it on the main root? Or will they also obey to the individual robots.txt in the other subdirectories?
Thanks in advance,
Brakkar
