Keeping spiders off co-branded sites

how to keep spiders from indexing our partner / co-branded sites

[BillingOrchard]

I run a company, Example, and we have numerous co-branded sites setup for resellers. They all use our same site, but are tracked by using a different dns A record. For example: widget.example.com tracks sales from our partner widget. When searching on Google for our primary keywords, we do show up as #3 and #1 respectively. Good thing sure, but the problem is that it's not our main site of www.example.com, but rather one of our partner sites, gadget.billingorchard.com.

How can I get Google and other spiders to only index our main site? While we appreciate our partners, these aren't really valid referrals. We'd obviously love to stay ranked well for those words.

Thanks,
Craig

[edited by: rogerd at 5:07 am (utc) on Dec. 23, 2004]
[edit reason] No specifics, please. [/edit]

[treeline]

Keeping Google, Yahoo, MSN out is easy. You want to use a robots.txt file on your site to disallow indexing of those areas.

The consequences may not be all positive. First, you'll lose showing up several times in the search results. Second, your affiliates won't show up at all, ever, in search engine results. This may not please them. They may even need to set up independent websites to cope. Then you'll be back where you started, with less control.

If you want to use robots.txt, a good guide is:

[robotstxt.org...]

[BillingOrchard]

Thanks for your response.

I don't want to block the spiders completely though from any files. I guess what I'm looking for is a way to tell the spiders, "only index files on www.billingorchard.com". Is that possible?

If I did something like check for the referring URL and then redirect to our main site if it's coming from Google, would that inhibit Googlebot from spidering our site? When spidering, I'd assume there's no referral from Google, correct?

Thanks again for your help.

[Jon_King]

Set up a duplicate set of pages, point your resellers to those pages and block them using robots.txt. The robots.txt will keep you from duplicate content and the resellers pages being indexed.

Your original pages will be (are) indexed and you should rank where you want with no one else pointing to them. Many manufacturers have completly different domains for reseller content for this very reason.