Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Keeping spiders off co-branded sites

how to keep spiders from indexing our partner / co-branded sites

8:15 pm on Dec 22, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 22, 2004
votes: 0

I run a company, Example, and we have numerous co-branded sites setup for resellers. They all use our same site, but are tracked by using a different dns A record. For example: widget.example.com tracks sales from our partner widget. When searching on Google for our primary keywords, we do show up as #3 and #1 respectively. Good thing sure, but the problem is that it's not our main site of www.example.com, but rather one of our partner sites, gadget.billingorchard.com.

How can I get Google and other spiders to only index our main site? While we appreciate our partners, these aren't really valid referrals. We'd obviously love to stay ranked well for those words.


[edited by: rogerd at 5:07 am (utc) on Dec. 23, 2004]
[edit reason] No specifics, please. [/edit]

11:12 pm on Dec 22, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 18, 2003
votes: 0

Keeping Google, Yahoo, MSN out is easy. You want to use a robots.txt file on your site to disallow indexing of those areas.

The consequences may not be all positive. First, you'll lose showing up several times in the search results. Second, your affiliates won't show up at all, ever, in search engine results. This may not please them. They may even need to set up independent websites to cope. Then you'll be back where you started, with less control.

If you want to use robots.txt, a good guide is:


9:05 pm on Dec 23, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 22, 2004
votes: 0

Thanks for your response.

I don't want to block the spiders completely though from any files. I guess what I'm looking for is a way to tell the spiders, "only index files on www.billingorchard.com". Is that possible?

If I did something like check for the referring URL and then redirect to our main site if it's coming from Google, would that inhibit Googlebot from spidering our site? When spidering, I'd assume there's no referral from Google, correct?

Thanks again for your help.

9:41 pm on Dec 23, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 16, 2002
votes: 1

Set up a duplicate set of pages, point your resellers to those pages and block them using robots.txt. The robots.txt will keep you from duplicate content and the resellers pages being indexed.

Your original pages will be (are) indexed and you should rank where you want with no one else pointing to them. Many manufacturers have completly different domains for reseller content for this very reason.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members