Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Restrict googlebot to access folder



1:19 pm on Jul 1, 2006 (gmt 0)

5+ Year Member

How do I restrict googlebot from accessing and caching all files in a folder called /secure on a server using modrewrite?


3:07 pm on Jul 1, 2006 (gmt 0)

5+ Year Member

I have done some research on my own, and found this syntax :

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*google.* [NC]
RewriteRule /theforbiddendirectory/(.*) - [F]

Would this work, and do i need to include the full directory path from the root of the server, or the root from the site itself?

Full directory path :

or :

I need to have this 100% correct, i only want to restrict google from 1 directory, not the whole site...


3:46 pm on Jul 1, 2006 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Looks like auto-generated code to me. For use in httpd.conf or conf.d:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} googlebot [NC]
RewriteRule ^/theforbiddendirectory/ - [F]

For use in .htaccess, remove the leading slash in the rewriterule pattern, so that "^/theforb" becomes "^theforb"

Then install and test your code. Wannabrowser is handy for a quick test run pretending you're Googlebot.

I presume that you have already told google not to spider this directory path by using robots.txt...



4:03 pm on Jul 1, 2006 (gmt 0)

5+ Year Member

Hi Jim,

Thanks for your answer. I suppose this piece of code goes into the .htaccess in the root directory. How can i simulate googlebot in wannabrowser?



4:40 pm on Jul 1, 2006 (gmt 0)

5+ Year Member

Sorry Jim, I allready found it. I needed to enter "googlebot" in the user agent (duh!)

Featured Threads

Hot Threads This Week

Hot Threads This Month