MSNbot lost our 301?

Any ideas?

[Marls]

Hi Guys,

All was well for several months with MSNbot and our 301 from oursite.com/?referrer=google (AdWords tracking URL) to oursite.com/

Then we noticed MSNbot was asking for, and getting, ourhomepage.com/\
Looked all over for a link to same and couldn't find one. Next thing we knew, MSNbot no longer asked for or got the 301, and instead now lists oursite.com/?referrer=google in the site search.

Have reported to MSN spam – don't know what else to do. Any ideas, hints. . .clues?

Thanks!

[Receptional]

Next thing we knew, MSNbot no longer asked for or got the 301, and instead now lists oursite.com/?referrer=google in the site search.

Either they do - or they don't - got a 301 message. It is not MSN's choice as to what result they are given, that destiny is in your own hands.

Sounds to me like someone upgraded the server and the settings got tweaked.

But since you brought this up - maybe you can answer me this related question: [webmasterworld.com...] since it sounds like you had it licked before it all went pear shaped for you.

Dixon.

[Marls]

Receptional --

Thanks so much for responding!

"Sounds to me like someone upgraded the server and the settings got tweaked"

WOW! We did go to new servers, but the 301s are still working correctly and look fine in a server header check.

Can you explain "tweaked"? What should I be looking for? And what's with MSNbot's search for oursite.com/\?

Here's what was working:

RewriteCond %{QUERY_STRING} referrer= [NC]
RewriteRule ^(.*)$ [oursite.com...] [R=301,L]

Sorry, don't know a thing about java, but can tell you that the referral string, along with the 301, shows-up in the logs.

Thanks!

[jdMorgan]

Marls,

This code looks strange. Not so strange that it wouldn't work for what you said you need, but it might redirect URLs you didn't intend it to.

For redirecting only your home page, or the index page of the directory in which this code resides, it should be:

RewriteCond %{QUERY_STRING} referrer= [NC]
RewriteRule ^$ http://www.example.com/? [R=301,L]

And to redirect any page on the site if requested with the referrer query appended:

RewriteCond %{QUERY_STRING} referrer= [NC]
RewriteRule (.*) http://www.example.com/$1? [R=301,L]

Receptional, your previous thread is locked due to age. The answer to your questions is that all requests to your server are logged. Even if you issue a 301 redirect, the log entry is created (and shows the 301 response). If the 301 is to your own site, then you should see a second request as the client responds to your 301 redirect. A browser will normally respond immediately by re-requesting the desired content from the URL you provide in the redirect response, whereas a robot may queue the redirected request for later processing.

Jim

[Marls]

Jim,

What a joy to see you respond! I've already learned so much from you just by "lurking" your other posts.

I'll give this a shot. Makes sense.

But here's the really odd thing -- Googlebot and Yahoo slurp have been following the 301s (as is/are) just fine. It's only MSNbot that's suddenly come unhinged -- go figure :)

Thanks!

[Marls]

Hey Jim,

Quick question. I just noticed for our non-www to the www the code reads:

RewriteCond %{HTTP_HOST}!^www\.oursite\.com [NC]
RewriteRule ^(.*)$ [oursite.com...] [R=301,L]

I also just noticed the \.com and don't know if that's a problem as well. If it's not, should the code read:

RewriteCond %{HTTP_HOST}!^www\.oursite\.com [NC]
RewriteRule ^$ [oursite.com...] [R=301,L]

Thanks so much!

[jdMorgan]

Quick question. I just noticed for our non-www to the www the code reads:

RewriteCond %{HTTP_HOST} !^www\.example\.com [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

That is correct, although I would remove the [NC] from the RewriteCond.

I also just noticed the \.com and don't know if that's a problem as well. If it's not, should the code read:

RewriteCond %{HTTP_HOST} !^www\.example\.com [NC]
RewriteRule ^$ http://www.example.com/$1 [R=301,L]

I'm sorry, I don't know what you are asking here. If you change the RewriteRule pattern to "^$", then only requests for your home page would be redirected to the correct domain.

Ultimately, it'll be up to you to understand and test this code thoroughly.

For more information, see the documents cited in the Apache forum charter [webmasterworld.com] and the tutorials in the Apache forum section of the WebmasterWorld library [webmasterworld.com]

Jim

[Marls]

Thankyou, thankyou!

[Marls]

Hi Guys,

Tried the new code and MSNbot took the 301 for /?referrer=google! However it still has yet to follow the 301 through to the home page, which Googlebot and Yahoo Slurp do perfectly.

And now on a site search in MSN we get: www.our-site.com/?referrer=google as the home page, whereas on a www.our-site.com search we get both this and the home page.

Any ideas? Thanks and here?s was we?re using:

RewriteEngine on

RewriteCond %{QUERY_STRING} referrer= [NC]
RewriteRule ^$ [our-site.com...] [R=301,L]

RewriteCond %{QUERY_STRING} source= [NC]
RewriteRule ^$ [our-site.com...] [R=301,L]

RewriteCond %{QUERY_STRING} ref= [NC]
RewriteRule ^$ [our-site.com...] [R=301,L]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*index\.html\ HTTP/
RewriteRule ^(.*)index\.html$ [our-site.com...] [R=301,L]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*index\.php\ HTTP/
RewriteRule ^(.*)index\.php$ [our-site.com...] [R=301,L]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*index\.htm\ HTTP/
RewriteRule ^(.*)index\.htm$ [our-site.com...] [R=301,L]

RewriteCond %{HTTP_HOST}!^www\.our-site\.com
RewriteRule ^(.*)$ [our-site.com...] [R=301,L]