Welcome to WebmasterWorld Guest from 188.8.131.52
A little digging revealed that a search for "www.mydomain.com" showed our domain on the 4th-5th page as well. (all our other sites come up #1 for that search).
A little tech info about the site:
1. It's on a dedicated server (had to, because the forums were killing the shared hosting account).
2. It's been redirecting non-www to www URLs via .htaccess since 2002 (based on advice received here)
3. PR has remained at 6, and all pages appear to be indexed (can't find many supplementals). Although the "site:www.mydomain.com" command returns roughly 3 times the number of pages we have.
4. Aside from using tried-and-true html standards (pages validate), we've never really SEO'd the site (like I said - just a hobby site).
5. Site's been using adsense to support it almost since it was released to the public.
6. Inbound links have actually increased since last July (all one-way). To include links from the Wall Street Journal & Fast Company, among others.
7. 95% of content is original (we do post some relevant press releases from time to time). New content is added daily (5-10 pages of articles) as well as at least 70-100 pages of forum posts (all of which eventually get crawled).
So.....now that you know all that, my question is...what could be causing the site to come up on the 4th-5th page for a "www.mydomain.com" search (which seems to be a symptom of what's hurting the site as a whole). We still enjoy good traffic from Yahoo/MSN, and receive some traffic from google (but only a shadow of it's former volume), and have never participated in reciprocal linking.
Is it a penalty of some kind? A technical glitch? Something that will eventually work itself out?
I just fixed this 2 weeks ago and still am on page 3 when I search for "www.mysite.com" - I am hoping this was the problem and eventually my site will be listed first for "site" search once google sees a correct 404 error for non-existant pages.
Just to be sure here, you never redirect to an error page. The error page is displayed at the URL of the original page request, and should have an HTTP status (in the HTTP header, sent before the HTML page itself) of 404.
The page should also have a <meta name="robots" content="noindex"> tag on it so that at its "real" location, of http://www.domain.com/errors/error.404.html it can never be indexed.
The crucial thing about the 404 error page is not what the user sees on their screen (that text can be customised to be anything you want, and is best done with some basic site navigation to send the user on their way) but the status code in the HTTP header when you request a URL that does not exist. Use a program like WebBug to check that out.
In Apache the custom error page is enabled in the .htaccess file using a command like:
ErrorDocument 404 /errors/error.404.html