why our homepage response different status code like 200,404,301?

HTMl response, status codes

[johnlim9988]

Hi,

why our homepage response different status code such as 200, 404, 301 to different request at different time?

Acturally the site homepage works well.

I think about it long time and cannot get the clue. Pls help.

details are as following,

Host: 209.191.87.214 - 09:21:33 26/Sep/2006 (Status: 200)
Request: GET / HTTP/1.0 Size: 37276 bytes
Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
/////////////////////////////////////
Host: 209.191.87.214 - 09:21:29 26/Sep/2006 (Status: 301)
Request: GET / HTTP/1.0 Size: 302 bytes
Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
/////////////////////////////////////
Host: 209.191.87.214 - 09:21:30 26/Sep/2006 (Status: 301)
Request: GET / HTTP/1.0 Size: 302 bytes
Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
/////////////////////////////////////
Host: 217.56.57.170 - 09:10:00 26/Sep/2006 (Status: 200)
Request: GET / HTTP/1.1 Size: 37286 bytes
Agent: Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
////////////////////////////////////

Host: 65.54.188.53 - 08:57:08 26/Sep/2006 (Status: 404)
Request: GET / HTTP/1.0 Size: 269 bytes
Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)

[theBear]

The 301 is easy enough to understand if there is a rewrite rule to redirect the non www version of the domain to the www version or the other way around.

Is it possible that the 404 occured after the page had been deleted say through a delete or rename operation and the new version was not yet on the server?

Is it also possible that there is a non renderable character after the / in the request that resulted in the 404?

[edited by: theBear at 3:10 am (utc) on Sep. 27, 2006]

[johnlim9988]

Yes, it is a non readerable character after the / in the request that resulted in the 404.

The non readerable character is %20, but I cannot understand why it has such strange request.

Does such strange request domain.com/%20 has any side effect to the site?

[hutcheson]

Presumably someone has a link to your domain that accidentally contains an extra space: http colon "//example.com " instead of "//example.com".

That's not a valid URL, but the IE tries to turn it into one, resulting in "//example.com/%20".

The bad link might be on your site, or on someone else's link to you.

[johnlim9988]

But we study the log file and cannot find the referer. Only the googlbebot always want to spider "domain.com/%20"

Such behavior has any side effect for the site?

[tedster]

No bad side effects as long as your server keeps saying 404.

[johnlim9988]

Here is another similar problem,

Beofore we have a links directory then we delete that directory about seven months ago. But googlebot/yahoo slurp/MSNbot always spider that directory until now.

Such behavior of robots have any side effect? Is this the reason MSN not cache our sites or google downgrade our site pages SERP? Should I ban the robots to spider that no exist directoy at robots.txt?

[theBear]

Any change in the link structure can have ranking implications. Hub scores get impacted, internal linking changes so PR flow gets changed.

But in and of itself removing one page should be a minor issue, if however it is many pages then it may take some time to get sorted out.

In addition to the ranking changes do to PR shifts, and hub scores, you will take a hit in your long tail distribution caused by the reduction in unique words and phrases on your site. The long tail distribution change could have major impacts on traffic.

The above is all "in theory" of course, other things could be in play that your question doesn't provide any clues about.

[edited by: theBear at 4:15 pm (utc) on Oct. 3, 2006]