Yahoo Indexing Problem

Has anyone noticed this?

[Rollo]

Our network has many URLs that end in / as in www.mydomain.com/page/ ...for some reason, Yahoo seems to be indexing things as www.mydomain.com/page in the SERPS and when one clicks they get a 404.

Does anyone else get this or is it just us?

[otc_cmnn]

/page

/page/

/page/default.html

should all return the same result - always. If it does not then your server is improperly configured.

Easy fix on your end .

naturally *domain.com/* will show as *domain.com* looks cleaner and _should_ return the same result.

I was recently removed from yahoo COMPLETELY for a similiar such small and seemingly innocuous technical problem.

[Rollo]

well, I use mod rewrite to make unfreindly php URls freindly, in this case, Yahoo is only reading page and not page/ and click throughs bring up pure 404s. Ironically, MSN picks up only page also (rather than page/), however these links work. I think it's a Yahoo bug, but we've rewrote some lines to make / or no / optional. It works but I'm not sure if it will cause future issues with dupe content. Knock on wood.

[jd01]

Rollo,

Better to redirect to the / than make it optional... depending on the structure of your pages/directories, you could use something like this:

RewriteRule ([^.]+[^/])$ /$1/ [R=301,L]

This will rewrite any URL that does not contain a .(dot) and does not end in a / to the same location with a /

Justin

[Rollo]

It's a perplexing problem. I wish Yahoo would just fix it. I'm worried about making the engnines think this is dupe content but it seems that both solutions will work in this regard... or do you think one way would cause an issue here? I image that if one person links to "page" and another to "page/" this could confuse the engines even though it's the same page. Maybe a redirect would work better... hmmmm.

[jd01]

Yes, usually a redirect is the best way to go.

this is a file on a domain:
http://yoursite.com/file

this is a directory on a domain:
http://yoursite.com/file/

this is a file on a sub-domain:
http://www.yoursite.com/file

this is a directory on a sub-domain:
http://www.yoursite.com/file/

They are 4 completely different, unique locations, and though they may (usually) contain the same information, they may also contain entirely different information. Therefore, SE's usually treat these *correctly* as different locations, and not an equal value that is expected to generate the same information.

So, yes, if you can access a single set of information in two different ways (URL paths) there is always the chance of duplicate content, because really, it is.

Justin

[Rollo]

Well, our code that deals with this is really some 300 lines and fairly complex, but here is the main section:

if((substr ($req, -1,1)!= '/')&&(substr ($req, -4,1)!=".")&&(substr ($req, -5,1)!=".")) $req.="/"; //It's added to solve / issue in url of Yahoo.
$sql="SELECT oldurl, newurl FROM #__redirection".
" WHERE oldurl".
" LIKE '".$req."'".
" LIMIT 1";
//die($sql);
$database->setQuery($sql);

It works beautifully, but I'm still worried that Google or MSN will take "page/" as not eqaul to "page" given this code and penalize me for dupe content. Unfortunatly, given the language and complexity of or SEO tools, we can't impliment some of the good sugesstions I've read.

[jd01]

I can't see your whole code/process, so obvoiusly, there is some speculation, but you should be able to do this straight from php with an if() and a 301 header if the location does not end in a / or contain a .(dot).

My guess is, you will need to find the right reaction to the failed test, but it should be possible. (EG do you redirect immediately, or are there variables you need to set/adjust first? You should be able to do either, as long as you don not 'serve' any information prior to the redirect header.)

$uri=getenv("REQUEST_URI");
if(eregi("^([^.]+[^/]{1})$", $uri)) { header ("HTTP/1.0 301 Moved Permanently"); header("Location: $uri/"); }

Hope this helps.

Justin