Welcome to WebmasterWorld Guest from 54.198.118.102

Forum Moderators: phranque

Message Too Old, No Replies

Am I doing it right?

Prevent Google to access one of my directory using robots.txt

     
8:24 am on May 15, 2003 (gmt 0)

New User

10+ Year Member

joined:Feb 27, 2003
posts:17
votes: 0


Hi..

I want to prevent google accessing one of my directory.. i have included the following syntax in robots.txt file... pls let me know its right!

User-Agent: Googlebot
Disallow: /<directory name to be blocked>/

8:46 pm on May 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 1999
posts:2141
votes: 0


Looks OK to me! :)

If you want to brush up a little, here's a handy tutorial [searchengineworld.com] and a validator [searchengineworld.com].

10:17 pm on May 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Will Google's Imagebot follow googlebot instructions, or do you need a seperate lne for that?

dave

11:44 pm on May 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


carfac,

Google's normal crawler UA is "GoogleBot"
The ua for imagebot is "GoogleBot-Image"

Therefore disallowing "GoogleBot" disallows both 'bots because of prefix-matching, while disallowing GoogleBot-Image only disallows the image spider.

Jim

11:46 pm on May 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


bashyam,

Note lowercase "a":


User[b]-ag[/b]ent: Googlebot
Disallow: /<directory name to be blocked>/

Jim
3:19 am on May 19, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Thanks Jim!

I was unsure.... I did not know if you had to match the bot name exactly...

dave

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members