Welcome to WebmasterWorld Guest from 50.19.34.234

Forum Moderators: phranque

Message Too Old, No Replies

Am I doing it right?

Prevent Google to access one of my directory using robots.txt

     

bashyam

8:24 am on May 15, 2003 (gmt 0)

10+ Year Member



Hi..

I want to prevent google accessing one of my directory.. i have included the following syntax in robots.txt file... pls let me know its right!

User-Agent: Googlebot
Disallow: /<directory name to be blocked>/

DaveAtIFG

8:46 pm on May 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Looks OK to me! :)

If you want to brush up a little, here's a handy tutorial [searchengineworld.com] and a validator [searchengineworld.com].

carfac

10:17 pm on May 17, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Will Google's Imagebot follow googlebot instructions, or do you need a seperate lne for that?

dave

jdMorgan

11:44 pm on May 17, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



carfac,

Google's normal crawler UA is "GoogleBot"
The ua for imagebot is "GoogleBot-Image"

Therefore disallowing "GoogleBot" disallows both 'bots because of prefix-matching, while disallowing GoogleBot-Image only disallows the image spider.

Jim

jdMorgan

11:46 pm on May 17, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



bashyam,

Note lowercase "a":


User[b]-ag[/b]ent: Googlebot
Disallow: /<directory name to be blocked>/

Jim

carfac

3:19 am on May 19, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Jim!

I was unsure.... I did not know if you had to match the bot name exactly...

dave

 

Featured Threads

Hot Threads This Week

Hot Threads This Month