Should I just ignore WMT page index count?

[AnthonysItalianFood]

OK, this has been driving me crazy! Since Panda, the amount of indexed pages shown in Webmaster Tools keeps dropping and dropping while the amount of actual indexed pages that I see in Google's index remain still indexed.

At present my sitemap information is as follows:
Submitted: 79 Indexed: 1 EEEik! Insert horror movie sound effect here.....

The indexed page count in WMT has consistently been dropping since April 2011 when Panda slapped me for no reason. I've done a million things and spent 1,000's of hours revising and cleaning up for the Panda and have made much progress, however, GWT just keep showing less and less pages indexed. But this is not the case when I use the stie: command and or just paste a specific url in Googles search field. All pages are in the indexed.

Here are some specifics with leaving my actual doamin name out of the picture:

site:domain.com = 264 results

site:www.domain.com = 28 results

site:domain.com/* = 71 results

site:www.domain.com/* = 4 results

15,274 backlinks to my site

In GWT>Site Cinfiguration>Settings> I have preferred domain set to domain.com (without the www) - I've had this set for a while since most indexed pages point to the non www domain.

I used to have a 99% page index status in GWT and all was happy.

So to sum all this up.

Do I just need to stop looking at GWT stats in regards to indexed pages and stop stressing? Or is there something else going on?

Thanks for any help on this.

[tedster]

Well, the first two data points you offer seem to point at a canonical problem with your website.

Beyond that, it does sound like the WMT sitemap numbers are buggy. I've seen something like this before and I sometimes think the data refers to "pages indexed BECAUSE of Sitemap crawling" instead of "Sitemap URLs that are indexed" - by any mechanism at all, including all the usual URL discovery mechanisms.

The site: operator often show bugs of its own. For example, if a site has five internal directories, then site:example.com may show considerably lower numbers than the total of all five results for site:example.com/dir1 + site:example.com/dir2... etc

[g1smd]

There's certainly a www/non-www problem that needs fixing.

You should certainly run Xenu LinkSleuth over your site, and twice: once starting at www.example.com/ and again starting at example.com/ - you might uncover some issues by doing that.

[AnthonysItalianFood]

Thanks for the response. Yes, I definitely think there is some canonical confusion going on. Just added a 301 redirect to my .htaccess in efforts to fix this:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]

So far seems to be working great!

Will see how things go.

@g1smd - "Xenu LinkSleuth"... Pretty slick tool there! Thanks for that!

[edited by: tedster at 6:50 pm (utc) on Mar 12, 2012]
[edit reason] make the example domain display [/edit]

[AnthonysItalianFood]

Glory Hallelujah!

now get this:

Sitemap - Mar 12, 2012 - submitted 70 - indexed - 69

Oh yes! Thanks for the help guys! :-)

Peace!

[g1smd]

Now that you have the canonical redirect in place, run Linksleuth again (temporarily untick the "check external links" option).

You're looking for any links within the site that point to the wrong version. You will need to correct all of those.


Make sure your htaccess rules are in the correct order. In particular be sure that no request can invoke an unwanted multiple step redirection chain. This means:
- the non-www/www redirecting rule should be the last of your redirects,
- redirects should all use RewriteRule syntax; do not use Redirect or RedirectMatch at all,
- external redirects should be listed before internal rewrites.

[AnthonysItalianFood]

@g1smd ran the linkslueth as you suggested again and "nothing" is pointing the www :-) Looking Good!

Thanks for the help. It is much appreciated.