Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: goodroi

Message Too Old, No Replies

How to block a subdomian

Where to put the file

     
5:51 pm on Dec 14, 2008 (gmt 0)

Full Member

10+ Year Member

joined:May 3, 2004
posts:297
votes: 0


Say the mobile version of our website lives on the subdomain mobile.example.com, with the files located at www.example.com/mobile

I'm not convinced that Google isn't punishing us for duplicate content, so I'd like to block the bots from this subdomain so as far as they're concerned, it doesn't exist.

We put a robots.txt file in www.example.com/mobile with the following:

User-agent: *
Disallow: /

It seems to have no effect. What should we do to disallow this subdomain?

11:08 pm on Dec 14, 2008 (gmt 0)

Administrator

WebmasterWorld Administrator coopster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 31, 2003
posts:12533
votes: 0


Looks correct to me.

It seems to have no effect.

Why do you say that?

2:33 am on Dec 15, 2008 (gmt 0)

Full Member

10+ Year Member

joined:May 3, 2004
posts:297
votes: 0


It looks like Google is still indexing our mobile pages, although I'm not sure how to see when the last visit by Googlebot was. Our webmaster tools are showing that our regular pages are restricted by Google's mobile crawl. Weird.
3:46 am on Dec 15, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


1) Do you do any User-agent or Accept header detection to redirect mobile clients to your mobile site?

2) Do you have a proper 301 redirect from the URLwww.example.com/mobile/<anything> to mobile.example.com/<anything> to prevent duplicate content problems?

Jim

[edited by: jdMorgan at 3:46 am (utc) on Dec. 15, 2008]

5:39 am on Dec 15, 2008 (gmt 0)

Full Member

10+ Year Member

joined:May 3, 2004
posts:297
votes: 0


jd-

1) We do have user agent detection, so if the client is on a mobile device, they should see the mobile version of our site.

2) Good question, and something it looks like we need to get in order. www.example.com/mobile/<anything> is not properly redirecting to mobile.example.com/<anything>. In fact, we don't have an .htaccess file in the mobile folder, which we probably should. This could be making a huge mess, as Google could be seeing the same content on 3 different pages:
www.example.com/<anything>
www.example.com/mobile/<anything>
mobile.example.com/<anything>

Could this be causing the bots to ignore robots.txt as well?

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members