Welcome to WebmasterWorld Guest from 54.211.136.250

Forum Moderators: goodroi

Message Too Old, No Replies

Several robots.txt?

   
1:01 am on Mar 14, 2004 (gmt 0)

10+ Year Member



Hello,
I wish to know if several robots.txt will work?

I don't plan to disallow all the directories I don't want to be crawled from the root (/www/robots.txt) robots.txt because I don't want someone to load this file and discover the urls of some directories.

So I plan to put several robots.txt additionally to the one in the root (/www/directory/robots.txt ....) .

Will it work? Does crawler only look for it on the main root? Or will they also obey to the individual robots.txt in the other subdirectories?

Thanks in advance,
Brakkar

1:18 am on Mar 14, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Only the robots.txt file in the root will be requested
www.*nameofyoursite.com/robots.txt

ncw164x