Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

solving 302 dupe issues on freehost site with robots.txt?

         

blut

5:13 pm on May 15, 2007 (gmt 0)

10+ Year Member



I apologize in advance if there is already a topic about this. I searched through many(!) pages and couldn't find something that addressed it directly.

Anyways, I have a domain name that points to a website hosted on a free webhosting company. Basically I purchased the domain name to have something consistent in case the free hosting company goes under and then I would be able to redirect the domain name elsewhere (to a new free host or the webspace provided by my ISP, for example) without informing everyone the URL has changed.

So the domain name got indexed really fast by Google and Yahoo. In the top 5 on the first page of the search in both. 5 months later, it vanished from Google. Its been gone from the index for about a month now. Yahoo and MSN still index it.

I wondered if this could be a duplicate content issue, as both the domain name AND the site on the free host were once indexed.

So, my question regarding the robots.txt is this:

If I put a robots.txt with the disallow "extension" in the root of the free host's directory to indicate to spiders to not index the free host website, will this somehow screw with the spiders that crawl my domain name purchased from godaddy? Is this a good idea or not really necessary? I thought this might be a way to tackle the potential dup content, but I am not certain this would have the desired effect.

The redirection is a 302. I know they are regarded as nasty, but I feel using a 301 would indicate that the page is permanently moved, which is not the case.

Thanks in advance for your help and advice!

[edited by: encyclo at 7:27 pm (utc) on May 15, 2007]
[edit reason] specifics, see terms of service [/edit]

tedster

5:53 am on May 17, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hello blut, and welcome to the forums.

If I understand you correctly, the actual files reside at the free server. The domain name you purchased only redirects to the home page on the free server, and it has no served content of its own. If you place a robots.txt file that disallows the files on the free server, then googlebot will not have any place where it is allowed to get the content.

So, do not place the robots.txt file on the free server. Whether the redirect is a 302 or a 301, I'd say the robots.txt disallow rule will wipe you out.

I suggest you get the domain name itself actually hosted somewhere. There are very low priced options available.

blut

12:11 pm on May 20, 2007 (gmt 0)

10+ Year Member



Thanks Tedster for your insight and advice! Greatly appreciated.