Forum Moderators: goodroi
We run a site with an american .com and a canadian .ca version of the site. Since it's the same content, etc.. and we don't want to get the duplicate content gods angry we've put a ban on all crawlers for the .ca site. It would appear that Yahoo's not obeying this and has our .ca site ranking for some obscure terms.
The file has been up for about 3-4 months now, so anything that was there has had plently of time to go away.
Does anyone have any suggestions for dealing with Yahoo not obeying this file? They claim to on their site, but the listings in the index say otherwise.
And in what form are your .ca URLs showing up in Yahoo search?
If you Disallowed them with robots.txt, and your .ca pages are showing up as URL-only listings, (possibly using link-text as the title) then that's to be expected, since they can include your URLs in their index by following incoming links (and the link text associated with those links) without actually fetching your pages.
There's also the possibility that you've got a syntax problem in your robots.txt file, but we can't be sure without a posted sample.
Jim
