homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

Restrict googlebot to access folder

 1:19 pm on Jul 1, 2006 (gmt 0)

How do I restrict googlebot from accessing and caching all files in a folder called /secure on a server using modrewrite?



 3:07 pm on Jul 1, 2006 (gmt 0)

I have done some research on my own, and found this syntax :

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*google.* [NC]
RewriteRule /theforbiddendirectory/(.*) - [F]

Would this work, and do i need to include the full directory path from the root of the server, or the root from the site itself?

Full directory path :

or :

I need to have this 100% correct, i only want to restrict google from 1 directory, not the whole site...


 3:46 pm on Jul 1, 2006 (gmt 0)

Looks like auto-generated code to me. For use in httpd.conf or conf.d:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} googlebot [NC]
RewriteRule ^/theforbiddendirectory/ - [F]

For use in .htaccess, remove the leading slash in the rewriterule pattern, so that "^/theforb" becomes "^theforb"

Then install and test your code. Wannabrowser is handy for a quick test run pretending you're Googlebot.

I presume that you have already told google not to spider this directory path by using robots.txt...



 4:03 pm on Jul 1, 2006 (gmt 0)

Hi Jim,

Thanks for your answer. I suppose this piece of code goes into the .htaccess in the root directory. How can i simulate googlebot in wannabrowser?



 4:40 pm on Jul 1, 2006 (gmt 0)

Sorry Jim, I allready found it. I needed to enter "googlebot" in the user agent (duh!)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved