Finally done but...

[cowcool]

I finally finished my custom 404.asp. When I checked non-existing documents using WebBug 5.3 from Aman Software with http 1.1 my IIS 5.0 setting, it returned "200 OK" header which is exactly what I want.

However, when checked with http 0.9 and 1.0 WebBug returned
400 Bad Request and HTTP/1.0 500 Internal Server Error.

What version of http protocol is googlebot currently using?. Will it be able to crawl my non-existing pages?. how about other major search engines?.

I just want to make my dynamic site to be crawable to major search engines, particularly Google and MSN

[duckhunter]

When you say dynamic, do you mean URL's with querystrings? If so, some spiders will choke and are unable to correctly follow dynamic links.

If you open your page with no parameters, as some spiders try, do you get a Server500 Error?

[cowcool]

Yes, I mean URL's with querystrings. I have converted those URL's into static URL's using asp server.transfer, i.e content.asp?id=5 into content5.htm and both Internet explorer and Netscape display them correctly so in theory, there should be no problem for spiders to follow those URL's.

I'm still trying to eliminate HTTP/1.0 500 Internal Server Error though. Any idea why does this error occur?.

[Dreamquick]

The problem may be that your utility is using a "pure" definition of the HTTP protocols - technically neither 0.9 or 1.0 support the HOST header athough in reality 1.0 happily accepts it.

If you are using a shared server then if you don't supply a HOST header your request will fail because the server doesn't know which site you are after.

I've never seen a HTTP/0.9 request so I'd suggest that 1.1 is the dominant form for browsers and some crawlers, with 1.0 being an alternative.

- Tony

[cowcool]

Thx Tony!