Forum Moderators: open
Deep Crawl has no problem with the redirect but Fresh just doesn't get it. Day after day it comes requesting pages with the www and not following the redirect except with robots.txt.
This is how i'm redirecting:
RewriteCond %{HTTP_HOST}!^mysite\.com
RewriteRule ^.*$ [mysite.com%{REQUEST_URI}...] [R=301,L]
I don't see how this could be a reason for a penalty or why some crawlers can't understand it or can't be bothered following it.
I reckon there's at least one link to you with the www. that freshbot is coming across on it's crawls. 301 redirect like I said in the second post and Fresh will soon get the idea.
I'm trying to figure out if there is a difference between using a 301 redirect and using CNAME. We have multiple domains (different ips) that point to our real site using CNAME.
When I plug in one of the redirecting urls to a search engine simulator it returns content from the real page. Is this seen as duplicate content (penalized by Google)? There are no links to the other urls.
TIA for any help/suggestions!
Dan
now I 301 redirect, and will always do so, cos, face it, most commercial sites ahve a bunch of domains ("second choice") set up for them.
SN