Forum Moderators: goodroi
I have 2 folders that I want to exclude from all search engine bots. In 1-hit.com I got this code for my robots.txt file. Is this correct?
#
# robots.txt generated by www.1-hit.com's robot generator
# Please, we do NOT allow nonauthorized robots any longer.
#
User-agent: *
Disallow: /main-folder/sub-folder/User-agent: *
Disallow: /my-folder/User-agent: URL_Spider_Pro
Disallow: /User-agent: CherryPicker
Disallow: /User-agent: EmailCollector
Disallow: /User-agent: EmailSiphon
Disallow: /User-agent: WebBandit
Disallow: /
........and so on........
Also, I wouldn't like various site copyers trying to copy my site. So I got this robots.txt code above, which excludes some of them. Now question: Won't I accidentally disallow google and other bots from spidering my whole site?
All I want is this:
To allow:
/main-folder/
but
-------------------
To disallow:
/sub-folder/, which is in /main-folder/
and
/my-folder/
from *ALL* SE spiders.
-------------------
And to disallow:
*My whole site*
from
All the robots that copy my website, but not SE spiders.
-------------------
So is that code above correct?
Thank You.