Welcome to WebmasterWorld Guest from 54.224.17.208

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Restrict googlebot to access folder

     
1:19 pm on Jul 1, 2006 (gmt 0)

5+ Year Member



How do I restrict googlebot from accessing and caching all files in a folder called /secure on a server using modrewrite?
3:07 pm on Jul 1, 2006 (gmt 0)

5+ Year Member



I have done some research on my own, and found this syntax :

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*google.* [NC]
RewriteRule /theforbiddendirectory/(.*) - [F]

Would this work, and do i need to include the full directory path from the root of the server, or the root from the site itself?

Full directory path :
/var/www/vhosts/mysites.com/httpdocs/forbiddendirectory

or :
/forbiddendir

I need to have this 100% correct, i only want to restrict google from 1 directory, not the whole site...

3:46 pm on Jul 1, 2006 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Looks like auto-generated code to me. For use in httpd.conf or conf.d:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} googlebot [NC]
RewriteRule ^/theforbiddendirectory/ - [F]

For use in .htaccess, remove the leading slash in the rewriterule pattern, so that "^/theforb" becomes "^theforb"

Then install and test your code. Wannabrowser is handy for a quick test run pretending you're Googlebot.

I presume that you have already told google not to spider this directory path by using robots.txt...

Jim

4:03 pm on Jul 1, 2006 (gmt 0)

5+ Year Member



Hi Jim,

Thanks for your answer. I suppose this piece of code goes into the .htaccess in the root directory. How can i simulate googlebot in wannabrowser?

thanks
pascal

4:40 pm on Jul 1, 2006 (gmt 0)

5+ Year Member



Sorry Jim, I allready found it. I needed to enter "googlebot" in the user agent (duh!)
 

Featured Threads

Hot Threads This Week

Hot Threads This Month