homepage Welcome to WebmasterWorld Guest from 54.237.213.31
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
Forum Library, Charter, Moderators: mademetop

General Search Engine Marketing Issues Forum

    
DNS redirect
Gert_Jan

10+ Year Member



 
Msg#: 4269 posted 11:48 am on Apr 8, 2004 (gmt 0)

Hey Guys,

I have a question. We have a client who has a website and also bought several other domains witch redirect trough a DNS to the primary website... Is this considered spam by google? Because we are preforming good in other SE...

Thanx in advance...

 

DoppyNL

10+ Year Member



 
Msg#: 4269 posted 11:52 am on Apr 8, 2004 (gmt 0)

if domain1.com, domain2.com and domain3.com all display the same site (ie: domainx.com stays in the adress bar) then it is considered spammy; as you've got the same content on 3 different places. (allthough most of the time 2 may not be listed and 1 of those will)

If domain1.com and domain2.com are redirected to domain3.com (ie: adress bar allways display domain3.com) then it's not considered spammy.

Robert Charlton

WebmasterWorld Administrator robert_charlton us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4269 posted 5:53 am on Apr 10, 2004 (gmt 0)

What Doppy said plus... it depends on how many redirects you have. Even if you're not creating mirrors, if you have too many domains redirecting via 301s to your main domain, there's a possibility the engines won't like it.

PS: I hadn't really read your title... "DNS Redirect"... before I posted. If you're talking about what I think you're talking about, which is a redirect from the registrar where the domain name is parked, it occurs to me that Network Solutions, eg, uses 302s. Not sure how Google would look at a lot of 302s point to your main domain.

brucec

10+ Year Member



 
Msg#: 4269 posted 2:34 pm on Apr 10, 2004 (gmt 0)

That's a good point. I personally I don't think GO Daddy counts. As big as Go Daddy is, I am sure the SE's understand that people do that, but I understand it as Meta tag refreshes from the domain itself.

Ah, I don't know any more. This whole thing is confusing. If someone can explain it to ME too, I would appreciate it.

Robert Charlton

WebmasterWorld Administrator robert_charlton us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4269 posted 8:44 pm on Apr 10, 2004 (gmt 0)

As big as Go Daddy is, I am sure the SE's understand that people do that

Do what?

Meta tag refreshes from the domain itself...

Meta refresh redirects are never a good idea. I'm talking about server redirects only.

Here's a good introduction:

An Introduction to Redirecting URLs on an Apache Server
For mod_rewrite beginners
[webmasterworld.com...]

Instead of mod_rewrite, which I still find intimidating, I've tended to use the simpler PermanentRedirect syntax in .htaccess.

I'd search the forums on Google for...

site:webmasterworld.com apache redirect syntax

- as well as for things like...

htaccess redirects
301 permanent redirects
PermanentRedirect
Redirect Permanent
redirect 301

etc.

claus

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4269 posted 11:21 pm on Apr 10, 2004 (gmt 0)

>> if you have too many domains redirecting via 301s to your main domain, there's a possibility the engines won't like it.

That must be recent. Are you sure about that? If yes, then how many is too many?

>> Is this considered spam by google?

AFAIK, there's a few webmasters out there who does this with hundreds or thousands of domains. I don't suppose that's the case here, so you should be okay doing that, as long as you make sure you have a 301 redirect in place.

This is the way to do it if you are hosting on an Apache server:

In the root folder on your main domain, save an ascii text file and name it ".htaccess" (remember the dot). The content of that file should be what is between the lines below:

----------------------------------------------
RewriteEngine on

RewriteCond %{HTTP_HOST} !^my-real-domain.com
RewriteRule (.*) http://my-real-domain.com/$1 [R=301,L]

----------------------------------------------

This rule will make sure that all incoming traffic will go to the proper domain regardless of which vanity domain was entered in the address line.

Replace "my-real-domain.com" with the domain name in both lines above.

If you prefer to use the "www." subdomain in front of the domain name, remember to write that in both lines as well.

Robert Charlton

WebmasterWorld Administrator robert_charlton us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4269 posted 5:04 am on Apr 11, 2004 (gmt 0)

>> if you have too many domains redirecting via 301s to your main domain, there's a possibility the engines won't like it.

That must be recent. Are you sure about that? If yes, then how many is too many?

claus - Check out this thread, where I got my feet wet in the subject. I probably should have posted it initially...

Pointing multiple domain names to main site without mirrors
[webmasterworld.com...]

WebGuerrilla, in msg #17, brings up the subject of how the engines will look at this. His code is basically the same as yours....

It works fine and is very inexpensive. But I will tell you that it is not completely risk free.

Recently, an unamed search engine empolyee was looking through my domain registrations and noticed I owned about 40 similar domains. When these domains were typed in, the all resolved to a single site. (using the method listed above).

That led to the employee telling me that if I didn't stop putting up duplicate sites, he'd have to take action. After explaining that these additional domains where used for print ads and general type-in traffic, and that they weren't actually sites and I didn't ever promote any of the other domains on the web,he seemed o.k. with it, but he did say that in a case like that, they would prefer the domains were on separate sites that had robots.txt exclusions set up.

I think the concern is that in a link analysis system, a 301 from a domain that existed at one time ultimately passes some rank to the new location. That being the case, there is the possibility to be abusive if you intentionally register 100's of expired domains and 301 them all to a single location.

That's not what I was doing and I don't think that is the intent of most dites that use a similar system, but the experience definitely got me thinking more about how stuff may look.

andreasfriedrich then follows up to elaborate on the code and to consider whether, for this purpose, a 303 might be a more appropriate redirect than a 301.

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4269 posted 5:39 am on Apr 11, 2004 (gmt 0)

You can have your cake, and eat it too, in this case. Here's a workable setup:

# Redirect all requests except robots.txt requests to our single "main" domain
RewriteCond %{HTTP_HOST} !^my-real-domain\.com
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule (.*) http://my-real-domain.com/$1 [R=301,L]
# Now take care of robots.txt requests
RewriteCond %{HTTP_HOST} !^my_real_domain\.com
RewriteRule ^robots\.txt /alternate_robots.txt [L]

Create the file "alternate_robots.txt" with the following contents:

User-agent: *
Disallow: /

This will satisfy the Google engineer's requirements while redirecting all of the alternate domains to your "main" domain name. All pages, images, scripts, etc., are redirected by the first rule, except for robots.txt. If a request is made for robots.txt in one of the alternate domains, then the second rule serves "alternate_robots.txt" instead, disallowing all robots from all files. Only if robots.txt is requested from the "main" domain will your actual robots.txt file be served (by default).

Jim

claus

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4269 posted 2:56 pm on Apr 11, 2004 (gmt 0)

Long post follows...


>> WebGuerrilla, in msg #17, brings up the subject of how the engines will look at this
>> "Recently, an unamed search engine ..."

That post is from January 2003. A lot has happened in SE technology since then but it appears to me, that once in a while these issues crop up never the less, most recently Angelfire [webmasterworld.com] and Godaddy [webmasterworld.com]. It seems they are often caused by big hosting firms messing something up.

Also, that post mentions no less than 40 "vanity domains" (the cases mentioned above certainly has more domains than that, probably thousands.) But then again, there's absolutely nothing wrong with that. Certain businesses have a lot of domains due to their nature (eg. firms in webdesign, hosting, or household names with lots of brands), and regardless of the function or content of these, this should not be considered "spamming" per se.

Personally i also hold a few domains and the question always arises about "what is the proper thing to do with them if they have no content yet?" I have chosen to redirect some of them to active domains (alternative spelling, mostly) and put a blank "this domain owned by" page on some others (as, imho, it does not make sense to redirect a "widget.com" type domain to a "gadget.org" type site). In both cases i use redirects, mostly of the 301 kind (a few uses other methods, for specific reasons).

To the best of my knowledge, the 301 redirect remains a good method to use. (..and imho, afaik, fwiw)

>> Here's a workable setup:

While i agree that it's the proper way to "do that thing", i really don't think it's the right thing to do ;) Let me explain:

I have not yet considered serving distinct robots.txt files for inactive or redirect domains, and much less disallowing robots from such domains. Here's my reason for not doing so:

First, i cannot get into my head that SE spiders should be so blatantly stupid (pardon my French) as to attempt to index the web without a basic understanding of the mechanics of that web (including the RFC's and protocols in the abovementioned thread). This is ment in a positive sense - i really think they try to follow the current practices and procedures (at least regarding the big SE's)

Hence, if i redirect, say the URL "widgetz.com/page.html" to the ressource "widgets.com/page.html" then that does not mean that there are two such pages. There is just one page and that page is on one domain only, it's not even close to being a duplicate. (Jim, you expressed the same thought in the thread referenced above)

If a SE finds a link to that page on the web, it should request the URL "widgetz.com/robots.txt" to determine if that page is indexable. It will then find out that the ressource it requests have been moved to another domain, and that it should read it there in stead. That's the interpretation: The ressource has been moved. It is the same ressource and it's valid for the URL requested, it's not a ressource for another URL (even though it resides at another location).

Having read the "widgets.com/robots.txt" the SE spider will find out that the ressource "/page.html" is permitted (as relative URLs is used in robots.txt context), and it should proceed to fetch that ressource. Doing that, it will find out that the ressource does not reside on the URL that it thinks, but at another one. Fine. Then (re)read the robots.txt for that other domain and find out that the ressource is still permitted at the new location.

As it has been moved permanently, there's no need to revisit the old URL. Except an infrequent visit once evey 6-12 months just to see if the domain has changed hands and has become active, perhaps. All references to the old URL in the index should simply be replaced by the new, until change is detected at the old domain.

The practical implications of this would be some sort of lookup table, i assume. It would mean two initial readings of the same robots.txt file, but after establishing the fact that the ressource has been moved no more double reading would be necessary, as the lookup routine would take care of that.

Result:

  • Right domain and ressource read and indexed
  • Initially a little overhead (one small file read twice)
  • Long term; less ressources spent by the SE - fresh and reliable index

Anyway, i'm not in a position to tell the engineers how to build search engines. All i can do is to do the right things, and then hope that they also do that, otherwise i can only ban them.

Still, consider the alternative:

You enable the rule in post #8 and with that, an alternative robots.txt that disallows all spidering of these URL's.

So, what does the SE spider do? First, it sees a link to that "domain-with-a-z" and tries to grab the robots.txt. Then it's told that it is not allowed to index that URL.

This implies (for Google at least):

  • The contents of the file is never fetched and read
  • The relation between the wrong and the right domain is never established
  • The wrong URL will appear in the SERPS as a "ghost URL" (no snippet, title, or cache)
  • It might even rank higher than your site if it has good inbounds
  • In some special cases (google bugs [webmasterworld.com]) it might even replace your real site in the SERPS

So, it might not be a good idea after all.

>> andreasfriedrich then follows up to elaborate

...he sure did ;) Quite informative posts he made there.

nakulgoyal

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4269 posted 5:07 pm on Apr 11, 2004 (gmt 0)

Very very interesting information!

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4269 posted 12:47 am on Apr 12, 2004 (gmt 0)

It seems they are often caused by big hosting firms messing something up.

Yup!

>> Here's a workable setup:

While i agree that it's the proper way to "do that thing", i really don't think it's the right thing to do.

Yes, I should point out that I would only recommend the specific technique I posted for the case where the "extra" domains had not already been spidered and/or promoted - I was speaking more to Robert Charlton's case - And the case where domains are registered as part of "Brand preservation" so that other's don't get them and try to ride your coattails. I would expect that most webmasters would also want to include more code to suit their specific needs.

Search engines (generally) seem to have enough problems just following the basic request/response protocols as specified by HTTP/1.x, and things get even more complicated when they get into the next level of processing -- trying to "associate" and untangle related domains and such. Not only is the logic more complex, but there is no standard that defines the right thing to do, so each engine goes their own way. As a result, we get 302 page-jacking and engines that take a year to drop 404 and 410'ed pages.

Anyway, I'll quit while this is still a comment post and not a rant... ;)

Jim

WebGuerrilla

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4269 posted 5:34 am on Apr 14, 2004 (gmt 0)

For the record, my comments in the previous post involved domains that had previously been registered. (expired domains). The reason behind the scolding I got had to do with exploiting the PR transfer provided by 301's.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved