One of the pages on my site was an accidental spider-trap. It was a calendar that could be surfed to the end of time. I have subsequently fixed the problem, restrained it to 1 years forward and 4 months back, for a total of 17 pages. I renamed the file so that the old file would no longer be there, and spiders would receive a 404 error. Slurp received a 404 for many months. But it still kept revisiting these pages. So, I hard-wired it so that it returns nothing but a permanent removal 410 code. It has been another several months.
Slurp just keeps revisiting and revisiting these pages. In fact, it revisits these pages more than any other page on my site. In fact today alone, out of 609 hits by Slurp, 92 of them have been on these pages. After a few months of 404 errors and a few more months of 410 errors, my question is does Slurp ever give up on files that are no longer there? What is the rationale?
