homepage Welcome to WebmasterWorld Guest from 54.196.201.253
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Please clarify this for the last time
Subfolder, particualr bots exclusion
damlag




msg:1526724
 10:12 pm on Jan 11, 2005 (gmt 0)

Hi,
I've read lots of posts and on the web, but would like you to help me out here for the last time.

I have 2 folders that I want to exclude from all search engine bots. In 1-hit.com I got this code for my robots.txt file. Is this correct?

#
# robots.txt generated by www.1-hit.com's robot generator
# Please, we do NOT allow nonauthorized robots any longer.
#
User-agent: *
Disallow: /main-folder/sub-folder/

User-agent: *
Disallow: /my-folder/

User-agent: URL_Spider_Pro
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: WebBandit
Disallow: /
........and so on........


I want the /main-folder/ to BE indexed by all search engines, I just don't want /sub-folder/ to be indexed. Is this correct? Won't my /main-folder/ be bypassed this way?

Also, I wouldn't like various site copyers trying to copy my site. So I got this robots.txt code above, which excludes some of them. Now question: Won't I accidentally disallow google and other bots from spidering my whole site?

All I want is this:

To allow:
/main-folder/

but
-------------------
To disallow:
/sub-folder/, which is in /main-folder/
and
/my-folder/

from *ALL* SE spiders.
-------------------
And to disallow:

*My whole site*

from

All the robots that copy my website, but not SE spiders.
-------------------
So is that code above correct?

Thank You.

 

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved