homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

want to ban MSNBot & Slurp from ONE dir only
I have the code...will it work?

Msg#: 3315 posted 9:30 pm on Apr 10, 2005 (gmt 0)

if I put this under directory to ban (them from), will it work? it works for the entire site.

<Directory /domain/www/offlimits>
AllowOverride AuthConfig FileInfo
Options +FollowSymLinks
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Yahoo!\Slurp [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Yahoo.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Slurp.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*msnbot.*$
RewriteRule .* - [F,L]

will they be able to access the rest of the site? I will ban on robots.txt too, but this is extra, in case they ignore robots.txt.

thanks in advance,



WebmasterWorld Senior Member 10+ Year Member

Msg#: 3315 posted 8:58 am on Apr 12, 2005 (gmt 0)

I would make sure I use a .htaccess in the directory I want to ban them from. This way the file is only invoked when a request is made that is within the directory and there is much less chance of banning them from the entire site.

EG /domain/www/offlimits/.htaccess

I believe the correct syntax is:

#RewriteCond %{HTTP_USER_AGENT} ^(.*)Yahoo(.*) [OR]
#RewriteRule (.*) - [F,L]

(No $ on the end of the condition.)

I would also consider [NC,OR] (No Case) and only 1 rule for each engine.



Msg#: 3315 posted 10:41 am on Apr 12, 2005 (gmt 0)

I want to ban SLURP from few directories (only)because my page rocks in Yahoo,if you ask me why i want to ban slurp from the "x" directories the answer is because the directory has pages that Yahoo does not like to be at the top 10 (if you are top and you are not a "friend" after a week you find your 10,000 pages site only the Index indexed in Yahoo. Because [of] that <snip> policy of yahoo i want someone to give me the right robots.txt that will not allow slurp and inktomi bots, because is not only slurp it is inktomi as well that <do this>.

[edited by: jdMorgan at 2:17 pm (utc) on April 12, 2005]
[edit reason] Removed off-topic comments. See TOS. [/edit]

Global Options:
 top home search open messages active posts  

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved