homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Keeping spiders off co-branded sites
how to keep spiders from indexing our partner / co-branded sites

 8:15 pm on Dec 22, 2004 (gmt 0)

I run a company, Example, and we have numerous co-branded sites setup for resellers. They all use our same site, but are tracked by using a different dns A record. For example: widget.example.com tracks sales from our partner widget. When searching on Google for our primary keywords, we do show up as #3 and #1 respectively. Good thing sure, but the problem is that it's not our main site of www.example.com, but rather one of our partner sites, gadget.billingorchard.com.

How can I get Google and other spiders to only index our main site? While we appreciate our partners, these aren't really valid referrals. We'd obviously love to stay ranked well for those words.


[edited by: rogerd at 5:07 am (utc) on Dec. 23, 2004]
[edit reason] No specifics, please. [/edit]



 11:12 pm on Dec 22, 2004 (gmt 0)

Keeping Google, Yahoo, MSN out is easy. You want to use a robots.txt file on your site to disallow indexing of those areas.

The consequences may not be all positive. First, you'll lose showing up several times in the search results. Second, your affiliates won't show up at all, ever, in search engine results. This may not please them. They may even need to set up independent websites to cope. Then you'll be back where you started, with less control.

If you want to use robots.txt, a good guide is:



 9:05 pm on Dec 23, 2004 (gmt 0)

Thanks for your response.

I don't want to block the spiders completely though from any files. I guess what I'm looking for is a way to tell the spiders, "only index files on www.billingorchard.com". Is that possible?

If I did something like check for the referring URL and then redirect to our main site if it's coming from Google, would that inhibit Googlebot from spidering our site? When spidering, I'd assume there's no referral from Google, correct?

Thanks again for your help.


 9:41 pm on Dec 23, 2004 (gmt 0)

Set up a duplicate set of pages, point your resellers to those pages and block them using robots.txt. The robots.txt will keep you from duplicate content and the resellers pages being indexed.

Your original pages will be (are) indexed and you should rank where you want with no one else pointing to them. Many manufacturers have completly different domains for reseller content for this very reason.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved