homepage Welcome to WebmasterWorld Guest from 54.211.73.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
Am I doing it right?
Prevent Google to access one of my directory using robots.txt
bashyam

10+ Year Member



 
Msg#: 2097 posted 8:24 am on May 15, 2003 (gmt 0)

Hi..

I want to prevent google accessing one of my directory.. i have included the following syntax in robots.txt file... pls let me know its right!

User-Agent: Googlebot
Disallow: /<directory name to be blocked>/

 

DaveAtIFG

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2097 posted 8:46 pm on May 15, 2003 (gmt 0)

Looks OK to me! :)

If you want to brush up a little, here's a handy tutorial [searchengineworld.com] and a validator [searchengineworld.com].

carfac

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2097 posted 10:17 pm on May 17, 2003 (gmt 0)

Will Google's Imagebot follow googlebot instructions, or do you need a seperate lne for that?

dave

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2097 posted 11:44 pm on May 17, 2003 (gmt 0)

carfac,

Google's normal crawler UA is "GoogleBot"
The ua for imagebot is "GoogleBot-Image"

Therefore disallowing "GoogleBot" disallows both 'bots because of prefix-matching, while disallowing GoogleBot-Image only disallows the image spider.

Jim

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2097 posted 11:46 pm on May 17, 2003 (gmt 0)

bashyam,

Note lowercase "a":

User[b]-ag[/b]ent: Googlebot
Disallow: /<directory name to be blocked>/

Jim

carfac

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2097 posted 3:19 am on May 19, 2003 (gmt 0)

Thanks Jim!

I was unsure.... I did not know if you had to match the bot name exactly...

dave

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved