Welcome to WebmasterWorld Guest from 54.226.130.194

Forum Moderators: goodroi

Message Too Old, No Replies

Several robots.txt?

     

brakkar

1:01 am on Mar 14, 2004 (gmt 0)

10+ Year Member



Hello,
I wish to know if several robots.txt will work?

I don't plan to disallow all the directories I don't want to be crawled from the root (/www/robots.txt) robots.txt because I don't want someone to load this file and discover the urls of some directories.

So I plan to put several robots.txt additionally to the one in the root (/www/directory/robots.txt ....) .

Will it work? Does crawler only look for it on the main root? Or will they also obey to the individual robots.txt in the other subdirectories?

Thanks in advance,
Brakkar

ncw164x

1:18 am on Mar 14, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Only the robots.txt file in the root will be requested
www.*nameofyoursite.com/robots.txt

ncw164x