Welcome to WebmasterWorld Guest from 54.159.50.111

Message Too Old, No Replies

Noindexing a subdomain I can't control?

     
2:37 am on Jul 22, 2013 (gmt 0)

Preferred Member

joined:Jan 12, 2012
posts:397
votes: 0


I'm working with a new vendor. Our technical agreement is setup like this:

I created a new cname record on my site. Now, a (previously unused) subdomain on my site points to a co-branded page on the vendor's site. The effect is that it looks like you're visiting my site when you hit the subdomain, but in reality the content you see on this subdomain is hosted by the vendor.

Now, the problem is that I do not want this content in Google's index since there isn't anything original about it. It aligns well with what my visitors might be looking for, and it puts some extra money in my pocket -- but the content isn't unique. I'm cool with that. Let me send my own visitors there, I don't need Google's. They'll have a good experience.

I can't control robots.txt on the subdomain, on account of my cname record. I also can't control the meta robots directive since the content is hosted on their servers.

Short of asking them to build a one-off noindex solution for me and me alone, is there any other way that I can prevent the pages on my subdomain from being indexed?
9:07 am on July 22, 2013 (gmt 0)

Moderator This Forum from GB 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Apr 30, 2008
posts:2507
votes: 140


Can't you check for HTTP_HOST of the request and if it is for a subdomain, rewrite robots.txt request to a different file where you block the crawling?
9:59 am on July 22, 2013 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


can you upload a .htaccess file to the root directory of that web server?
4:07 pm on July 22, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12708
votes: 244


Double-checking here, because it sounds like a configuration I met recently on an unrelated site:

Your primary domain is example.com

Your vendor is at spam.example.com Well, maybe not literally, but work with me here

The primary domain is hosted on your own server, or at least in your own space within shared hosting

The subdomain is physically located in a completely different place, so requests for the subdomain will never see the primary domain's htaccess and/or config.

Is that the setup?
4:21 pm on July 22, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member planet13 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:June 16, 2010
posts:3796
votes: 28


I would just ask the vendor to do a one-off noindex for the subdomain.

I am surprised that other websites haven't asked for that as well, with Panda and everything.
6:26 pm on July 22, 2013 (gmt 0)

Preferred Member

joined:Jan 12, 2012
posts:397
votes: 0


Double-checking here, because it sounds like a configuration I met recently on an unrelated site:

Your primary domain is example.com

Your vendor is at spam.example.com Well, maybe not literally, but work with me here

The primary domain is hosted on your own server, or at least in your own space within shared hosting

The subdomain is physically located in a completely different place, so requests for the subdomain will never see the primary domain's htaccess and/or config.

Is that the setup?


This is exactly right.
10:59 pm on July 22, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:July 19, 2013
posts:1097
votes: 0


It's pretty simple for them to noindex the whole thing.
All they have to do is put the following in the .htaccess of the subdomain:

Header set X-Robots-Tag "noindex"
11:38 pm on July 22, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12708
votes: 244


That's assuming that the two of you-- OP and vendor, I mean-- remain on good terms. But that's a whole nother discussion, having to do with whether it's ever a good idea to let some other person have control of a subdomain under your domain. Seems like there could be calamities in both directions.
12:25 am on July 23, 2013 (gmt 0)

Preferred Member

Top Contributors Of The Month

joined:May 24, 2012
posts:648
votes: 2


To answer it literally, you could:

a) run varnish (or a similar proxy) on a new ip address on your host.

b) change the A record for the subdomain to point to this new ip address.

c) put an entry in /etc/hosts on your host that points the subdomain to their server.

d) within varnish, configure them as the backend, but inject the X-Robots-Tag. Varnish makes injecting http headers easy.

However, if this is the company that I think it is, their whole model for this setup depends on organic search. In fact, a standard part of their contract requires that you point links from your domain to several pages of the subdomain...for this very reason.

Are you sure they have some way to drive traffic other than organic search?
2:34 am on July 23, 2013 (gmt 0)

Preferred Member

joined:Jan 12, 2012
posts:397
votes: 0


Yeah, I'm not feeling this setup. Might just have to do some API development and maintain control of everything

Without going into too many details, my partner is related to a classifieds vertical of which I have a ton of informational content.

I don't expect them to send me any traffic. I send my visitors to these pages through search forms on my site.
1:44 pm on July 23, 2013 (gmt 0)

Preferred Member

joined:Jan 12, 2012
posts:397
votes: 0


Thanks everyone for your help. Unfortunately, none of your suggestions will work for my situation. I stayed up all night working with their API and will be taking that route instead so I maintain control.

The stuff we do for Google...