homepage Welcome to WebmasterWorld Guest from 54.237.134.62
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to block a subdomian
Where to put the file
Musicarl




msg:3807246
 5:51 pm on Dec 14, 2008 (gmt 0)

Say the mobile version of our website lives on the subdomain mobile.example.com, with the files located at www.example.com/mobile

I'm not convinced that Google isn't punishing us for duplicate content, so I'd like to block the bots from this subdomain so as far as they're concerned, it doesn't exist.

We put a robots.txt file in www.example.com/mobile with the following:

User-agent: *
Disallow: /

It seems to have no effect. What should we do to disallow this subdomain?

 

coopster




msg:3807486
 11:08 pm on Dec 14, 2008 (gmt 0)

Looks correct to me.

It seems to have no effect.

Why do you say that?

Musicarl




msg:3807565
 2:33 am on Dec 15, 2008 (gmt 0)

It looks like Google is still indexing our mobile pages, although I'm not sure how to see when the last visit by Googlebot was. Our webmaster tools are showing that our regular pages are restricted by Google's mobile crawl. Weird.

jdMorgan




msg:3807589
 3:46 am on Dec 15, 2008 (gmt 0)

1) Do you do any User-agent or Accept header detection to redirect mobile clients to your mobile site?

2) Do you have a proper 301 redirect from the URLwww.example.com/mobile/<anything> to mobile.example.com/<anything> to prevent duplicate content problems?

Jim

[edited by: jdMorgan at 3:46 am (utc) on Dec. 15, 2008]

Musicarl




msg:3807630
 5:39 am on Dec 15, 2008 (gmt 0)

jd-

1) We do have user agent detection, so if the client is on a mobile device, they should see the mobile version of our site.

2) Good question, and something it looks like we need to get in order. www.example.com/mobile/<anything> is not properly redirecting to mobile.example.com/<anything>. In fact, we don't have an .htaccess file in the mobile folder, which we probably should. This could be making a huge mess, as Google could be seeing the same content on 3 different pages:
www.example.com/<anything>
www.example.com/mobile/<anything>
mobile.example.com/<anything>

Could this be causing the bots to ignore robots.txt as well?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved