Welcome to WebmasterWorld Guest from 18.104.22.168
The challenge with using a log analyzer is that entries into the logs are dependant upon the page being requested, if pages on my site receive no traffic (human or bot) these pages would not be able to be counted.
I am curious if anyone else has had this problem and what you did to solve it?
Any tracking procedure will as you say get only pages that have been requested.
A link follower will not be able to deal with dropdown lists (I could be wrong, if so I want to know what link tracker will do it.)
My concerns about an estimate is that I could be 10-15% off and this could have huge implications on a financial model you build on top of this number.
On top of the financial model it would be very valuable for me to know the actual number of pages on my site as this will help show how well search engines are indexing us.
I am surely not with the only organization who has had this problem as I can only imagine that most larger companies suffer from this issue. Specifically if you have blogs, community, and directory and thus page growth is occuring from users as apposed to a centralized development cycle.
At this point I am open for any kind of an idea, even brain storming.
But doesn't your CMS generate page counts for you? Most do.
I will update the thread with the solution when/if we come up with one.