Welcome to WebmasterWorld Guest from 126.96.36.199
I want to prevent google accessing one of my directory.. i have included the following syntax in robots.txt file... pls let me know its right!
User-Agent: Googlebot Disallow: /<directory name to be blocked>/
If you want to brush up a little, here's a handy tutorial [searchengineworld.com] and a validator [searchengineworld.com].
Google's normal crawler UA is "GoogleBot" The ua for imagebot is "GoogleBot-Image"
Therefore disallowing "GoogleBot" disallows both 'bots because of prefix-matching, while disallowing GoogleBot-Image only disallows the image spider.
Note lowercase "a":
User[b]-ag[/b]ent: GooglebotDisallow: /<directory name to be blocked>/
I was unsure.... I did not know if you had to match the bot name exactly...
Become a Pro Member