Hi, I've read lots of posts and on the web, but would like you to help me out here for the last time.
I have 2 folders that I want to exclude from all search engine bots. In 1-hit.com I got this code for my robots.txt file. Is this correct?
# # robots.txt generated by www.1-hit.com's robot generator # Please, we do NOT allow nonauthorized robots any longer. # User-agent: * Disallow: /main-folder/sub-folder/
User-agent: * Disallow: /my-folder/
User-agent: URL_Spider_Pro Disallow: /
User-agent: CherryPicker Disallow: /
User-agent: EmailCollector Disallow: /
User-agent: EmailSiphon Disallow: /
User-agent: WebBandit Disallow: / ........and so on........
I want the /main-folder/ to BE indexed by all search engines, I just don't want /sub-folder/ to be indexed. Is this correct? Won't my /main-folder/ be bypassed this way?
Also, I wouldn't like various site copyers trying to copy my site. So I got this robots.txt code above, which excludes some of them. Now question: Won't I accidentally disallow google and other bots from spidering my whole site?
All I want is this:
To allow: /main-folder/
but ------------------- To disallow: /sub-folder/, which is in /main-folder/ and /my-folder/
from *ALL* SE spiders. ------------------- And to disallow:
*My whole site*
All the robots that copy my website, but not SE spiders. ------------------- So is that code above correct?