Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Funneling crawlers from subdomains to your main website?



12:06 am on Apr 12, 2007 (gmt 0)

5+ Year Member

We have a very successful main website as well as 100+ partners who pay us for cobranded (custom header) yet otherwise identical versions of our main website. For example say our main website is www.ourdomain.com, then a partner would have companyname.ourdomain.com and the only difference is their header, the content is all the same.

We don't allow crawlers to crawl the cobranded versions of the site because of the duplicate content issues (100 cobrands X 500,000 pages) but at the same time we're leaving thousands of inbound links from some of the largest websites out there on the table.

We were discussing the possibility of "funneling" crawlers to our main website via 301 redirect. So for example if a crawler goes to somepartner.ourdomain.com/somepage we would instead 301 redirect the crawler to www.ourdomain.com/somepage (the definitive URL for the page and content). The only difference in the page is that it would no longer have the partners header on top. In theory this would allow those inbound links to count towards our main website.

The question is, would this be considered a high or low risk form of cloaking?



1:25 am on Apr 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

It's not cloaking at all. Even the most rabid cloaking opponent would have to admit that the same content is shown to all visitors.


4:32 pm on Apr 12, 2007 (gmt 0)

5+ Year Member

But it is delivering different content based on the user agent. The real user would get to see the cobrand [somecompany.ourdomain.com...] and the GoogleBot/Slurp/etc would get a 301 redirect to [ourdomain.com....] Isn't that considered cloaking?


8:50 pm on Apr 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

No. Cloaking is different content served on the same URL.

Keeping spiders from indexing duplicate content is actually recommended.


Featured Threads

Hot Threads This Week

Hot Threads This Month