HTTP/1.1 404 Not Found returned for URLs with hash tag

[AnnRick]

Hi,

I have this kind of URL structure in my magento website.
http://www.example.com/category-page/product-page/#21

But when I check this URLs in [webconfs.com...] it returns "HTTP/1.1 404 Not Found". I am not able to find the cause of 404 issue. This pages are working fine in browser though it shows 404 header in that tool. Will google consider it as 404 pages?

Can anyone help me with that? Any help in finding solution is highly appreciated.

Thanks
Ann

[edited by: phranque at 7:53 pm (utc) on Jun 16, 2014]
[edit reason] exemplified domain [/edit]

[penders]

This pages are working fine in browser...

You see the correct content, but what is the HTTP status code? 404? 200?

When you see a 404 returned in these tools, what is the content body? A 404 error document, or the correct content?!

A fragment identifier (everything after the #) is unlikely to cause a 404, since this is never actually sent to the server.

[AnnRick]

Tool returns below o/p:

HTTP/1.1 404 Not Found =>
Date => Mon, 16 Jun 2014 06:51:03 GMT
Server => Apache/2.2.27 (Unix) mod_ssl/2.2.27 OpenSSL/1.0.1e-fips DAV/2 mod_bwlimited/1.4
X-Powered-By => PHP/5.4.29
Expires => Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma => no-cache
Set-Cookie => frontend=2a9658d0666854adfddad7ba48c910f5; expires=Mon, 16-Jun-2014 07:51:04 GMT; path=/; domain=www.domain.com; HttpOnly
Vary => Accept-Encoding
Connection => close
Content-Type => text/html; charset=UTF-8

URL is working fine in browser and loading correct data but not in tool. Will this affect on SEO as I have lots of this kind of URL in my website.

[penders]

URL is working fine in browser and loading correct data...

But what is the HTTP status code returned to the browser? A page can "look" OK and still return a "404 Not Found" status.

It is possible that your site is returning a 404 only to this (and related) tools, however, it is also possible that your site is (incorrectly) returning a 404 for all requests. If this is the case and Googlebot is also seeing a 404 then yes, this will affect SEO.

In Google Webmaster Tools (GWT) are erroneous 404's being reported?

[AnnRick]

But what is the HTTP status code returned to the browser? A page can "look" OK and still return a "404 Not Found" status. => Yes. Page is OK only that tool is returning 404 status code.

Google webmaster tool is not showing any crawling error for that kind of URLs.

[penders]

Google webmaster tool is not showing any crawling error for that kind of URLs.

That is promising.

...what is the HTTP status code returned to the browser?

In Google Chrome you should check the "Network" tab in the Object Inspector (Ctrl + Shift + I). There is a similar tool in Firefox.

[AnnRick]

Everything looks OK in Google Chrome - Network tab.

Thank you for such a quick and useful response.

[lucy24]

Hypothesis: The tool's robot is crawling from a blocked range-- very common with online tools of all description when they're not big enough to have their own IP range. At some time in the past, you cleverly decided that serving a 404 is a more effective approach than the usual 403. (This really does work better with some robots: "Darn, I guess they don't have one" as opposed to "Hm, what is it they don't want me to see?")

So the 404 they report is the 404 they see. But as long as both humans and search engines can get in, all is well.

The fact that this question was posted in the php forum means you were thinking of another possibility. It's common for a request to return 200 all the time, including when there's no content to be served-- in the worst case, when no page has been built at all. It's at least theoretically possible for the opposite to happen: a fully formed page accompanied by a 40x header. Good to know this wasn't the case here.

[phranque]

welcome to WebmasterWorld, AnnRick!

A fragment identifier (everything after the #) is unlikely to cause a 404, since this is never actually sent to the server.

penders is correct here.
the browser uses the fragment identifier.
the request sent to the server contains everything up to but not including the pound sign. (#)

URL is working fine in browser and loading correct data but not in tool. Will this affect on SEO as I have lots of this kind of URL in my website.

using the "fetch as googlebot" feature in GWT will give you an indication of how googlebot sees this url.