homepage Welcome to WebmasterWorld Guest from 54.167.75.155
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to exclude from crawling subdomain
exclude the crawling subdomain
fendy




msg:3248506
 3:02 pm on Feb 10, 2007 (gmt 0)

I have a problem. I want to prevent search bots to index all the pages from subdomain.example.com because that is not a unique content and I am afraid that it will harm our rankings.

The text of those pages is something many sites have it on their sites so it is definately a dupliacte content.

Can I used the robots.txt file

and how to do it please

thank you

[edited by: tedster at 3:21 pm (utc) on Feb. 10, 2007]
[edit reason] use example.com, remove specifics [/edit]

 

goodroi




msg:3250548
 9:56 pm on Feb 12, 2007 (gmt 0)

upload a robots.txt file to subdomain.example.com/robots.txt

to block all robots the robots.txt should have only the 2 following lines in it:
User-agent: *
Disallow: /

fendy




msg:3253021
 3:37 am on Feb 15, 2007 (gmt 0)

But do you think we should do it , will it harm our site if we would not?

goodroi




msg:3253290
 1:00 pm on Feb 15, 2007 (gmt 0)

so it is definately a dupliacte content

i'm not a believer of having duplicate content on a site. it is generally possible with some extra work to make it original content. rewrite it, add new stuff or reformat it. the search engines will filter or penalize duplicate content, from my pov it means the same = no traffic. my personal preference is to block duplicate content if i can't make it unique content.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved