Timing out during crawling

Why does it happen?

[tkarade]

Why do some urls time out when crawling a site?
What does it mean when they time out?

Thanks,
TK

[wilderness]

The pages don't load fast enough or the bot has limit settings on how long it waits for page delivery.

"206" Incomplete page.

[tkarade]

Thanks wilderness,

if you don't mind, could you elaborate a little more on your answer.

1. The pages don't load fast enough:
-> what are the reasons for this to happen

2. The bot has limit settings on how long it waits for page delivery:
-> Are these settings specific to the bot or can they be change in another way (say through the operating system)

Thanks,
TK

[wilderness]

1. The pages don't load fast enough:
-> what are the reasons for this to happen

Could have either too many images, images too large, entire page images and content tottal too large, even a troubled image which your site host is not serving up complete. Another example might be too much content. I had a page which was cut and pasted (it has since been removed) from an MS-Access file the page was 180kb and hardly any bot returned a 200 result on the page.
I'm inclined to think simialr things happen on the multitude of Flah and other media pages which reach near a meg in size :(

2. The bot has limit settings on how long it waits for page delivery:
-> Are these settings specific to the bot or can they be change in another way (say through the operating system)

Neither you or I have any control over bot settings (algorithms.)