Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Several robots.txt?

1:01 am on Mar 14, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 17, 2003
votes: 0

I wish to know if several robots.txt will work?

I don't plan to disallow all the directories I don't want to be crawled from the root (/www/robots.txt) robots.txt because I don't want someone to load this file and discover the urls of some directories.

So I plan to put several robots.txt additionally to the one in the root (/www/directory/robots.txt ....) .

Will it work? Does crawler only look for it on the main root? Or will they also obey to the individual robots.txt in the other subdirectories?

Thanks in advance,

1:18 am on Mar 14, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
votes: 0

Only the robots.txt file in the root will be requested