How to track search result rankings over time

[bigtoga]

So we're a small two-person web dev team responsible for about a 3000-page website. I have struggled with how to maintain a historical view of our search result rankings. I read the threads here when someone says, "How do I diagnose a penalty? I noticed that today we have what appears to be a -50 penalty." When I read these types of threads, I think to myself, "But how do you know? How is it that they know that a *single page* has this penalty?" We have so many pages that I am lost in trying to remember what page ranked where.

We do have about 30 pages that want to monitor ranking on a "very often" basis, another 150 that we want to monitor on a "less often" basis, and then the rest on an "infrequent" basis. But I don't know how to do this.

I'd appreciate some suggestions on the "how" of tracking such info. I just want to be able to know when a penalty has been enforced. Right now, I'm in the dark. I've tried Excel and manually doing it but that is just not working.

[edited by: tedster at 6:14 pm (utc) on May 19, 2009]

[tedster]

I have never bothered trying to guess the various searches any particular page might rank for, because what matters is what searches are really bringing in traffic. So it's server log analysis that is the primary and most useful tool. And server log analysis tools will show you what searches are bringing in the traffic.

We don't discuss specific Google tools by name here (see the Google Forum Charter [webmasterworld.com]) but you probably know that Google hates automated rank checking. To do it on a regular basis means taking some care not to bombard Google's server with frequent automated requests. Yoou can easily get your IP shut out by Google if you do, and in some cases I've heard of, where Google feels there was serious abuse of their system, there seem to have beem penalties placed against an associated domain.

There are many commercial rank checking and SEO products available that keep keep records over time for you, and you can find these tools through the obvious searches. But it is important not to obsess too much about ranking. Any given check may not show you what other searchers are seeing anyway these days. As I mentioned above, a better performance indicator to focus on is actual traffic, as seen in your server logs. If that goes wobbly, then rankings are a good diagnostic tool.

[bigtoga]

Thanks, tedster.