| 7:03 am on Sep 23, 2013 (gmt 0)|
what is the problem you are trying to solve?
if a requested url doesn't exist it is appropriate to return a 404 response (or 410 in some case) so not all 404s are a problem.
| 7:23 am on Sep 23, 2013 (gmt 0)|
phranque thanks for your prompt answer :)
Actually a week back I was reading this somewhere that we should mention 404 pages in robots and block the SEs. So I got confused and raised the question here.
Btw if there are so many pages then what we should do ? Most of the times We see it in Google Webmaster Tools for some sites with loads of pages.
| 9:07 am on Sep 23, 2013 (gmt 0)|
Are you asking about the 404 page itself? Sure, slap a "noindex" meta on it if you're worried. But I don't think search engines index error documents; they probably don't even read them.
:: detour here as I realize I've got a very easy way to check ::
I have a stylesheet that's used only by my error documents; almost nobody sees it except humans who got locked out by mistake. afaik the googlebot has never asked for it-- that's what I just detoured to look up-- though they routinely get my other stylesheets. That can only be because they don't even look at the 404/410 page, but just note the header. (The plainclothes bingbot constantly asks for the file, but that's just part of the act.)
Don't block error documents in robots.txt; that could be counterproductive. Besides, robots don't normally ask for your 404 page unless there's been a mistake of some kind. They get handed it whether they want it or not.
| 9:20 am on Sep 24, 2013 (gmt 0)|
Lucy24...can you pls. explain a little more about your stylesheet. Hope you have no problem :)
| 7:25 pm on Sep 24, 2013 (gmt 0)|
You don't really need to know about the stylesheet. I just mentioned it as something I use for information.
All my error documents-- 403, 404, 410-- call on a single stylesheet that isn't used by anything else. Humans landing on an error page will automatically get the css-- and also the favicon or apple-touch-icon, if they haven't already got it. Robots don't normally get stylesheets. But the major search engines do, because they index them along with everything else. So my thinking is that if google has ever physically seen any of my custom error pages, they would have learned that the css exists and they would eventually ask for it.
When your server sends out a 4xx response, it automatically sends out the appropriate error document. But a robot can choose not to look at the document.
If you don't use a separate stylesheet for error documents, and you don't have a "noindex" header, you could try an exact-text search for something from your error document and see if it turns up. This will of course only work if there's something unusual about the wording.
But we still haven't nailed down the original question. Are you asking how to keep search engines from indexing your 404 page? Or are you asking something entirely different?
| 11:24 am on Sep 25, 2013 (gmt 0)|
Ok. Actually I wanted to if your site got lots of 404 products Pages which is no longer required then what do you do to lessen the numbers.
| 11:36 am on Sep 25, 2013 (gmt 0)|
404s aren't necessarily a bad thing - they tell people (or search engine bots) that the page they're looking for doesn't exist at that location.
If the page genuinely doesn't exist, then its ok (and best practice) to continue to serve a 404, however if the page *does* exist and you've changed its url, then you should set up a 301 redirect from the old url to the new one.
| 5:17 pm on Sep 25, 2013 (gmt 0)|
if a previously valid url is no longer being served it is appropriate to provide a 410 Gone response.
these 410s will eventually drop from the GWT list.
| 9:01 pm on Sep 25, 2013 (gmt 0)|
o/t but I have to add this since I brought up the subject earlier:
After saying categorically that the googlebot has NEVER asked for my error-document stylesheet-- implying that they don't know it exists-- only yesterday I found them asking for it. (Technically Monday, but I discovered it yesterday.) Coincidence, sure, but a slightly creepy one.
| 7:22 am on Sep 26, 2013 (gmt 0)|
Thanks phranque , I think that's what I wanted to know. Thanks to all for their time to answer.