Does Deepbot clean out the index of pages that do not exist anymore or that have been moved?
Oaf357
12:26 am on Apr 16, 2003 (gmt 0)
I sure hope it does. I've recently restructured my site.
Visi
2:14 am on Apr 16, 2003 (gmt 0)
Have to say no at this time. Have commented before on this and am cnvinced that this is now a function of freshbot. Seems that the deep bot just crawls and then during the following month, the freshbot confirms the 404 errors. Have seen this cycle over the past couple of months, and tends to be 8-10 weeks after the deep crawl for the database to shed all the old links. On the other hand these links may not be displayed. but freshbot seems to be one cycle behind the deepcrawl (maybe even 2? cycles)as to which database it is confirming to as far as 404 errors. Just our observations on what we have been seeing in our logs from a major site update last year. Also noticed that Slurp is also rehitting the 404's extensively the last 12 weeks. Yes our 404 page is correctly serving up a 404 code. In this area since the announcement with yahoo it has become repetative and frequent. Not quite sure what that means yet, but no doubt here both engines trying ways to get database current. Just my 2 cents worth.
mcavic
2:22 am on Apr 16, 2003 (gmt 0)
Hmm. I haven't had any 404's on my site lately, but I 301'd about a dozen pages. For me, when Freshbot sees the 301, it gives up and doesn't even request the real page. Deepbot correctly replaced my old pages with new during the last full crawl, though.
Visi
2:40 am on Apr 16, 2003 (gmt 0)
Let me clarify my post...listings seem to be updated in the past 3 months, after what I saw as a problem with the listings not being updated for a number of months. However freshbot is still looking for the pages, indicating to me that google although updating results, is checking for a period of time underneath the listings...referencing some previous crwal that these pages are actually gone. Something new for the freshbot in the last couple of months. Have sen the pattern for last 2-3 months, and quite regular, deepcrawl, freshbot visits, then for 2-3 days looks at previous 404 listings. Becoming less frequent each month, suggesting that after a certain amount of confirmations these are removed? Just for the record these pages were removed last september to october timeframes, but google IMHO had real issues late last year in getting a good updated database, without the 404's being listed. They crawled the new pages and added them but left obsolete pages on the listing. Much better from Feb on listings. People on this board had indicated that updates should occur in a deepcrawl cycle...we had waited 4 cycles. Whatever the issue seems much better now. As I mentioned seeing a tremendous amount of activity by slurp bot, and much of this also seems to be in old pages. However they are still after 3 months attempting to visit the long lost pages? What I am pointing out here is that this bot is not spidering, but confirming, to an existing database of some sort. Some major changes in the approaches by both of these bots.
Jesse_Smith
2:49 am on Apr 16, 2003 (gmt 0)
bah, according to Google, I got 2,510 dead links listed! I changed some URLs back in November/December, from .html to .shtml! (To find out how many dead links you got listed, make a 404 error redirect to a certian URL, then enter the title of the error page in quotes and wait for the next dance.)
I thought Google hated redirects? Sure doesn't look like it!
