Forum Moderators: open
Do I just wait, hoping it goes away? I could set a crawl-delay in my robots.txt file but if I made it really long I'm worried I could hurt my Yahoo rankings.
Months ago I RewriteRule'd all traffic for a specific directory to the specific file that Yahoo is grabbing, so that's part of the mystery. But not all of it. There were only a few dozen files in the old directory but Yahoo is requesting them over and over and over and then getting redirected to the new file, even though I used a 301 so Yahoo shouldn't keep requesting the old URL's. But anyway, I disallowed access to the old, non-existent directory with robots.txt, so maybe that will take care of it.
Sorry to clutter up the forum. Would have removed my original post if I saw a way to do that....