Site on page 4 for "www.example.com" - Why?

[WebFusion]

Last July, one of my hobby sites that had been enjoying great traffic for years (since late 1999) suddenly saw a huge dropoff. Previously to that day, the site had had roughly 4000 pages in the top 10 (mostly #1's) for the relevant phrases on google, but all dropped back to the 4th-5th page.

A little digging revealed that a search for "www.mydomain.com" showed our domain on the 4th-5th page as well. (all our other sites come up #1 for that search).

A little tech info about the site:

1. It's on a dedicated server (had to, because the forums were killing the shared hosting account).

2. It's been redirecting non-www to www URLs via .htaccess since 2002 (based on advice received here)

3. PR has remained at 6, and all pages appear to be indexed (can't find many supplementals). Although the "site:www.mydomain.com" command returns roughly 3 times the number of pages we have.

4. Aside from using tried-and-true html standards (pages validate), we've never really SEO'd the site (like I said - just a hobby site).

5. Site's been using adsense to support it almost since it was released to the public.

6. Inbound links have actually increased since last July (all one-way). To include links from the Wall Street Journal & Fast Company, among others.

7. 95% of content is original (we do post some relevant press releases from time to time). New content is added daily (5-10 pages of articles) as well as at least 70-100 pages of forum posts (all of which eventually get crawled).

So.....now that you know all that, my question is...what could be causing the site to come up on the 4th-5th page for a "www.mydomain.com" search (which seems to be a symptom of what's hurting the site as a whole). We still enjoy good traffic from Yahoo/MSN, and receive some traffic from google (but only a shadow of it's former volume), and have never participated in reciprocal linking.

Is it a penalty of some kind? A technical glitch? Something that will eventually work itself out?

[crazybrain]

I had a similar experience happen to 2 of my sites. I did notice that I had improperly handled my custom error pages. I had redirected all errors to full_url/error404.html - but later learned that I should redirect to /error404.html (without the full url). This way googlebot will see a true 404 for missing pages. I checked my logs and noticed several old pages getting redirected to my custom error page. Maybe google gave some kind of duplicate penality.

I just fixed this 2 weeks ago and still am on page 3 when I search for "www.mysite.com" - I am hoping this was the problem and eventually my site will be listed first for "site" search once google sees a correct 404 error for non-existant pages.

[WebFusion]

No....that's not it for us. We use a standard 404 page.

[Dayo_UK]

Webfusion

What is the PR on the non-www (you will probably have to use a tool to determine this as obv. any page request will redirected to the www)?

[g1smd]

>> later learned that I should redirect to /error404.html <<

Just to be sure here, you never redirect to an error page. The error page is displayed at the URL of the original page request, and should have an HTTP status (in the HTTP header, sent before the HTML page itself) of 404.

The page should also have a <meta name="robots" content="noindex"> tag on it so that at its "real" location, of http://www.domain.com/errors/error.404.html it can never be indexed.

The crucial thing about the 404 error page is not what the user sees on their screen (that text can be customised to be anything you want, and is best done with some basic site navigation to send the user on their way) but the status code in the HTTP header when you request a URL that does not exist. Use a program like WebBug to check that out.

In Apache the custom error page is enabled in the .htaccess file using a command like:

ErrorDocument 404 /errors/error.404.html

[crazybrain]

Thanks g1smd,

I did not add the no "noindex" to the error html. I will do that right away.