I would make sure I use a .htaccess in the directory I want to ban them from. This way the file is only invoked when a request is made that is within the directory and there is much less chance of banning them from the entire site.
Msg#: 3315 posted 10:41 am on Apr 12, 2005 (gmt 0)
I want to ban SLURP from few directories (only)because my page rocks in Yahoo,if you ask me why i want to ban slurp from the "x" directories the answer is because the directory has pages that Yahoo does not like to be at the top 10 (if you are top and you are not a "friend" after a week you find your 10,000 pages site only the Index indexed in Yahoo. Because [of] that <snip> policy of yahoo i want someone to give me the right robots.txt that will not allow slurp and inktomi bots, because is not only slurp it is inktomi as well that <do this>.
[edited by: jdMorgan at 2:17 pm (utc) on April 12, 2005] [edit reason] Removed off-topic comments. See TOS. [/edit]