Welcome to WebmasterWorld Guest from 54.159.50.111

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Restrict googlebot to access folder

     
1:19 pm on Jul 1, 2006 (gmt 0)

New User

5+ Year Member

joined:June 29, 2006
posts:17
votes: 0


How do I restrict googlebot from accessing and caching all files in a folder called /secure on a server using modrewrite?
3:07 pm on July 1, 2006 (gmt 0)

New User

5+ Year Member

joined:June 29, 2006
posts:17
votes: 0


I have done some research on my own, and found this syntax :

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*google.* [NC]
RewriteRule /theforbiddendirectory/(.*) - [F]

Would this work, and do i need to include the full directory path from the root of the server, or the root from the site itself?

Full directory path :
/var/www/vhosts/mysites.com/httpdocs/forbiddendirectory

or :
/forbiddendir

I need to have this 100% correct, i only want to restrict google from 1 directory, not the whole site...

3:46 pm on July 1, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Looks like auto-generated code to me. For use in httpd.conf or conf.d:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} googlebot [NC]
RewriteRule ^/theforbiddendirectory/ - [F]

For use in .htaccess, remove the leading slash in the rewriterule pattern, so that "^/theforb" becomes "^theforb"

Then install and test your code. Wannabrowser is handy for a quick test run pretending you're Googlebot.

I presume that you have already told google not to spider this directory path by using robots.txt...

Jim

4:03 pm on July 1, 2006 (gmt 0)

New User

5+ Year Member

joined:June 29, 2006
posts:17
votes: 0


Hi Jim,

Thanks for your answer. I suppose this piece of code goes into the .htaccess in the root directory. How can i simulate googlebot in wannabrowser?

thanks
pascal

4:40 pm on July 1, 2006 (gmt 0)

New User

5+ Year Member

joined:June 29, 2006
posts:17
votes: 0


Sorry Jim, I allready found it. I needed to enter "googlebot" in the user agent (duh!)