404 response: how's it treated by SEs?

How do bots treat pages returning 404 response

[No_Blonde]

I'm not sure whether it's the most appropriate forum. Still I've got a question. If a bot gets 404 response and custom 404 error page will it follow the links and index all the content which is on the page? How will it treat it? Any ideas?

[getxb]

When bots receives 404/410 response when trying to crawl a page, it simply excludes the page (and its content/links/etc.) i.e. that page doesn't get included in the refresh of the index.

[Receptional]

That's not entirely true. That's what they should do, I'll agree, but there are many occassions when 404s are in the serps and - as I understood it a year or two ago - I was told by one of the majors that they did not automatically exclude a page that gets a 404 until they are confident it was not simply a connectivity problem.

The other way in which a 404 page can remain in an index is if the page has not actually been CRAWLED by the bots, but has been picked up as a link from other pages. At that point, the search engines have not established that the link IS a 404 yet.

[getxb]

Lots to learn! Thanks for the info.

[Receptional]

Don't trust me... :)