How many times will a spider try to download a page, before it gives up and moves to fresh pastures, is it just the once, or will it try a couple of times then hit the roaad :)
Craig
lazerzubb
9:06 am on Nov 21, 2002 (gmt 0)
I think it depends on what code it get's back from the server, if the server is down, and the spider can't get no response, it will probably try a little bit later (GoogleGuy reported increases support for re-spidering of sites which had problem during the normal crawl)
I'm not sure how it is with the Fresh Crawl, i would guess that if it can't reach the page, it just continues, but not sure.
