Forum Moderators: open

Message Too Old, No Replies

Slurp is hammering one of my pages

~2000 requests for a single page daily for the last few days

         

MichaelBluejay

10:07 pm on Aug 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Slurp has been making ~2000 requests a day for the last few days to a specific deep page within my site. I have no idea why. It's really skewing my stat reporting, and I don't have an easy way to filter out Slurp from the stats, which are generated automatically by my host. I wrote to the form on Yahoo's help pages and got the predictable unhelpful form letter reply.

Do I just wait, hoping it goes away? I could set a crawl-delay in my robots.txt file but if I made it really long I'm worried I could hurt my Yahoo rankings.

MichaelBluejay

10:15 pm on Aug 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I should have checked my raw log files before posting....

Months ago I RewriteRule'd all traffic for a specific directory to the specific file that Yahoo is grabbing, so that's part of the mystery. But not all of it. There were only a few dozen files in the old directory but Yahoo is requesting them over and over and over and then getting redirected to the new file, even though I used a 301 so Yahoo shouldn't keep requesting the old URL's. But anyway, I disallowed access to the old, non-existent directory with robots.txt, so maybe that will take care of it.

Sorry to clutter up the forum. Would have removed my original post if I saw a way to do that....