Forum Moderators: open
Well done Yahoo!
That's odd - my experience with Slurp has been contrary... I had a client who had decided to use stealth domain forwarding (thanks, registrar companies!) for several domains (which Yahoo promptly spidered, creating some ugly instances of duplicate content) - when I corrected the issue and parked the domains and then set 301 redirects over to the primary domain Google and MSNBot picked up promptly... however...
... one month later and Slurp is still attempting to spider these domains, receiving the 301 Redirect code, and then trying again every day.
Has anyone else seen any similar Slurp mis-behaviour on domain name redirects..?
In just noticed MSNBot also picked up my 301, but Google still hasn't. I'm surprised.
It could very well be that pages containing a non-permanent redirect are stored in a location other than the main spidering index, and although they are now receiving the correct response, will probably remain in the a secondary location until a set period of time has passed, and therefore will be requested from the original location until that time expires.
Not sure, but this would make sense, and explains why redirects are picked up at different intervals for different sites. EG if the page has not been cached in 3 weeks and the original location is set to expire in 30 days, the redirect will appear to be picked up quickly, but if the page was cached 2 days ago, it will take almost a month for the effects of the redirect to stop a SE from requesting a page.
Justin
I just posted on a similar case.
'Yahoo, meet 404 and 301'
