how to disable others redirected to my site

redirected mod_rewite

[jollibee]

I have a site, like www.example.com. Someone has another site, like www.others.com. if the user visit www.others.com, the site redirected to my site, the url in the browser changes to www.example.com. How can I disable this.

If the user visit www.others.com, i will display a static page to tell the user to use the correct url to visite www.example.com.

Any tips? Thanks.

[JamieBrown]

Hi Jollibee. You could look at the referrer string and for those URLs where it contains www.others.com, show your message. This is very easy in any scripting language like PHP or ASP. This would only work for those users that do send referrer information, but it might get the message across to the owners of others.com.

Alternatively, you could try getting in touch with the hosting provider of www.others.com and let them know - try a WHOIS lookup on that domain. The hosting provider *might* do something about it, even though there's no obligation for them to do so.

Finally, you might just leave it - you are getting free traffic?

[jollibee]

The owner of www.others.com do that, and i don't want that happens.

"You could look at the referrer string and for those URLs where it contains www.others.com, show your message."

referrer string, you mean, http_refer?

Thanks Jamie

[janharders]

yes, HTTP_REFERER

you could also do it using mod_rewrite, just put something like this in a .htaccess

RewriteCond %{HTTP_REFERER} !^http://www\.others\.com
RewriteRule .* /messagegoeshere.html [L]

[jdMorgan]

Check for a positive, not a negative match, and for use in .htaccess, you must add a loop-stopper. Also, you should rewrite only "page"-type URL-paths, and not images, css, etc. Example:

RewriteCond %{HTTP_REFERER} ^http://(www\.)?others\.com
RewriteCond $1 !^your-message-page-here\.html$
RewriteRule ^(.+\.html)$ /your-message-page-here.html [L]

You will need to completely-flush your browser cache before testing each case in this code.

Jim

[jollibee]

I tried this, but it doesn't work.

When i visit www.others.com, it return "301 moved permanently", and the browser visit www.example.com when received the 301.

[janharders]

try adding
Options +SymLinksIfOwnerMatch
RewriteEngine On

before the rules in the .htaccess

btw - you're running apache as a webserver, right? It won't work like that on others.

[jollibee]

I add this to apache config file, but it doesn't work. Just wondering can i get HTTP_REFERER in this case. When a browser visit www.others.com, the site return "301", and then the browser visite the url in the 301 response( to my site, www.example.com ). The browser will add HTTP_REFERER in the second request?

<VirtualHost _default_:80>
ProxyRequests off

Options +SymLinksIfOwnerMatch
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://www\.others\.com/(.*)$ [NC]
RewriteRule .* http://www.example.com/redirect/domain_jump.html [L]

RewriteLog logs/meatball_rewrite_log
RewriteLogLevel 0

...
</VirtualHost>

[janharders]

mh, yeah, that's browserspecific. I guess, most won't, you're right. Just to be sure, test it yourself.
Use firefox and the "Live HTTP Headers" addon. Get it from [addons.mozilla.org...]

Open it, then visit www.others.com and you will see exactly what they send to your browser and what your browser sends to your site.

[jollibee]

This is the live http headers:

[others.com...]

GET / HTTP/1.1
Host: www.others.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-ecve: 300
Connection: keep-ecve

HTTP/1.x 301 Moved Permanently
Date: Fri, 25 Jul 2008 11:13:10 GMT
Server: Apache/2.0.52 (Unix)
Location: http://www.example.com/
Content-Length: 238
Keep-ecve: timeout=5, max=300
Connection: Keep-ecve
Content-Type: text/html; charset=iso-8859-1
----------------------------------------------------------
http://www.example.com/

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-ecve: 300
Connection: keep-ecve
Cookie: ec_apache_id=172.16.**.192.72031713509689.9; __cookie_client_data__=HEh8JXBg2BsY5bS7w4zwRs8J1Pal/HHe68QEYDTH/JEYjvc2cwDvtIjsACgBpVlcCIMtm9ryd/Ne9SAlbgqn2sioYwds9dZy; ec_apache_track=; ystat_cn_ss_488177=22_17867566_592976912; ec_apache_sid=172.16.60.195.00862878276476.0¦1216986162; _intl_temp_other_=w0ER0RhW1plXntR/97cJxJ/rwza32M8hvMwCJAF/EMNGXJhOQjdtBhT9tWE1xmjD; ec_apache_tracktmp=; _others_session=BAh7ByIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoMY3NyZl9pZCIlNGRmYjUwOTk3ZmZjOWE0MjJk%250AY2IzN2FjODk5MTNlNGE%253D--dacd19605daca78f61ce0ede18e5879cbf437763
If-Modified-Since: Fri, 25 Jul 2008 09:53:39 GMT
If-None-Match: "4889a2a3-fb69-20f05f"

HTTP/1.x 304 Not Modified
Date: Fri, 25 Jul 2008 11:13:14 GMT
Connection: Keep-ecve
Keep-ecve: timeout=5, max=100
Etag: "4889a2a3-fb69-20f05f"
Set-Cookie: ec_apache_sid=172.16.**.195.00862878276476.0¦1216986194; path=/; domain=.example.com

[edited by: jdMorgan at 11:53 am (utc) on July 25, 2008]
[edit reason] Obscured IP address [/edit]

[janharders]

yupp, no referer-header, it won't work. sorry.

got to find another way to stop them. question is, could redirecting even be considered illegal or something a court would stop?

[jollibee]

I compare two request, one use others.com, one use www.example.com, the header of the two requests is the same.

Just the client ask that. I have to told him, there is no way to do that :-).

The request from others.com:
http://www.example.com/

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-ecve: 300
Connection: keep-ecve
Cookie: ec_apache_id=172.16.**.192.72031713509689.9; __cookie_client_data__=HEh8JXBg2BsY5bS7w4zwRs8J1Pal/HHe68QEYDTH/JEYjvc2cwDvtIjsACgBpVlcCIMtm9ryd/Ne9SAlbgqn2sioYwds9dZy; ec_apache_track=; ystat_cn_ss_488177=28_17868470_592976912; ec_apache_sid=172.16.60.195.00862878276476.0¦1216987066; _intl_temp_other_=w0ER0RhW1plXntR/97cJxJ/rwza32M8hvMwCJAF/EMNGXJhOQjdtBhT9tWE1xmjD; ec_apache_tracktmp=; _ecxueyuan_session=BAh7ByIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoMY3NyZl9pZCIlNGRmYjUwOTk3ZmZjOWE0MjJk%250AY2IzN2FjODk5MTNlNGE%253D--dacd19605daca78f61ce0ede18e5879cbf437763

The request from example.com:
http://www.example.com/

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-ecve: 300
Connection: keep-ecve
Cookie: ec_apache_id=172.16.**.192.72031713509689.9; __cookie_client_data__=HEh8JXBg2BsY5bS7w4zwRs8J1Pal/HHe68QEYDTH/JEYjvc2cwDvtIjsACgBpVlcCIMtm9ryd/Ne9SAlbgqn2sioYwds9dZy; ec_apache_track=; ystat_cn_ss_488177=26_17868329_592976912; ec_apache_sid=172.16.60.195.00862878276476.0¦1216986994; _intl_temp_other_=w0ER0RhW1plXntR/97cJxJ/rwza32M8hvMwCJAF/EMNGXJhOQjdtBhT9tWE1xmjD; ec_apache_tracktmp=; _ecxueyuan_session=BAh7ByIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoMY3NyZl9pZCIlNGRmYjUwOTk3ZmZjOWE0MjJk%250AY2IzN2FjODk5MTNlNGE%253D--dacd19605daca78f61ce0ede18e5879cbf437763

[edited by: jdMorgan at 11:54 am (utc) on July 25, 2008]
[edit reason] Obscured IP address [/edit]

[jollibee]

Anyway, thanks janharders.

[jdMorgan]

There are other options. For example, you could deny access, and use a custom 403 error page to give your message.

The way I do this is to rewrite the request (using mod_rewrite in my main .htaccess file) to a special subdirectory. In that subdirectory, I create a .htaccess file that defines the 403 error page for that subdirectory (using ErrorDocument). Then I use mod_access Allow and Deny directives to forbid access to any file in that special subdirectory except the 403 errordocument itself.

So, any request that is rewritten to that subdirectory gets a "special" 403 error page, different from the main 403 error page used for the rest of the site.

There are a couple of applications for this technique. First, in cases like yours, you can send a customized error document based on the client request's HTTP_Referer, IP address, etc. And second, for known bad 'bots and abusive/malicious visitors, you can send a very short or even 0-byte 403 message body to save on wasted bandwidth.

This technique might be useful to you, depending on exactly what it is that you hope to accomplish.

Jim

[janharders]

jdMorgan: it still won't help with the deny-by-referer-case, since the request looks like an ordinary request or (even worse) www.others.com still ranks in search engines, the browser will just pass through the referer and he'll just see the serp-referer. blocking on that basis would be a painful thing to do

[jdMorgan]

Many search referrals carry the domain of the page listed in the SERPs as part of the query string. In that case, it's not at all painful to check to see if it is your own site or a different site. Unfortunately, none of the server request headers posted above show the Referer header, so they're not illustrative of the point, but detecting them isn't difficult.

As with any HTTP Referer-based access control, this method is not 100% effective, since the Referer is an optional header. But often it can be "effective enough."

Jim

[janharders]

yeah, it's possible the serp-referers show them --- but, google's don't, unless you included it in your query (or have I missed something) and that throws out a big chunk of them (or, in germany, basically all of them). But how do the serp-refs include the domain? I mean, some are using click tracking, ok, but they'll relocate the client via http headers, not a meta refresh, thus having the same problem, you get the serp as a referer, not the clicktracking nor www.others.com

which engines do fix that?

[jollibee]

"jdMorgan: it still won't help with the deny-by-referer-case, since the request looks like an ordinary request or (even worse) www.others.com still ranks in search engines, the browser will just pass through the referer and he'll just see the serp-referer. blocking on that basis would be a painful thing to do "

This is my question. There is no evidence for distinguish the two requests.

[jdMorgan]

jollibee,

In order to distinguish the two requests, you have to have requests in which the HTTP Referer: header was included. If the client does not send a referrer, then you cannot distinguish them... Sometimes clients send a referrer, and sometimes they don't -- It is up to the client (and the 'user') whether to send the Referer: header. And in some cases, even if the client sends the Referer: header, it can be blocked or removed by an intervening corporate or ISP proxy, over which the user has no control.

janharders is right, in that much of the time, even if you have the referrer, there still isn't enough information to make a decision. However, if you can catch just a few of these unwelcome referrers, then you can sometimes "make enough trouble" for the Webmaster of the site that is redirecting to you to make him remove the redirect.

As it stands, be grateful this is a 301 redirect (which tells the search engines to "remember" *your* URL). If it were a 302, then you might be in the mess that was discussed at length last year here on WebmasterWorld, with 302 "page-hijacking."

Jim

[jdMorgan]

One more note: When testing and trying to find differences, completely flush your browser cache, and delete all cookies for both sites -- "other" and yours. That way, you will see a 'fresh' request as from a 'new user.'

Jim