Google & Webmasterworld.com

Why doesn't google index all WebmasterWorld.com pages?

[vibgyor79]

Hey everybody,

Am I right in assuming that Google does not index all webmasterworld.com forum pages? If yes, why? I see that there are 1000s of plain html files/topics within WebmasterWorld.com but none of them are listed in Google.

If I want to implement a forum in my website, which is the best discussion board software, from Google's "point of view"? YaBB (yet another bulletin board - YabbForum.com)? PHPBB (PHPBB.com)? Something else?

Would Google list my forum pages?

Google doesn't seem to have any problems listing forum postings from WWWBoard (scriptarchive.com) bulletin board system

Ashwin

[ciml]

Google seems quite happy to list a few thousand results [google.com].

[rogerd]

Yeah, if Brett ever starts making blue fuzzy widgets he's gonna make a ton of money with those rankings! ;)

[vibgyor79]

LOL! That answers my question - I need BestBBS !

[Brett_Tabke]

Ya, there is something in the Google algo that limits the total number of pages indexed on any system. Who can blame them?

I also made many changes here in june that reworked many urls. The total number of potential static urls went up about 3 fold due to the new multipart thread system. I also got rid of all the cgi links that were laying around on the page - which meant some mass 404's for Google these last few crawls.

[danny]

Is the WW system free software? I notice it's been steadily improved, which suggests it's at least modifiable.

I'm thinking of putting some discussion boards onto my own site.

[rcjordan]

>WW system free software?

The buzz is that it's going to be "Enterprise Level" pricing ...if Brett ever gets around to canning it.

[mack]

Well lets face it, we all put it through it's paces here on Webmasterworld and it stands up very well. Think of the number of users,posts,pages and searches it delivers every day... very good and reliable software. :)

[Brett_Tabke]

Two other threads about it:
[webmasterworld.com...]
[webmasterworld.com...]

[vitaplease]

vibgyor79,

I was also wondering about that in this thread [webmasterworld.com] . Mainly because Google would be the ideal way to find old subjects/threads.

As far as I can see, there are about 235.000 messages (see the sum at the bottom of the forum index; [webmasterworld.com...] ).

I have never heard Brett post the total number of threads, but in this thread: [webmasterworld.com...] Brett mentions average number of messages and threads a day. Lets say 12 messages per thread as average over the years.

That would mean nearly 20.000 threads or 75% are missing at this moment in the Google index.

Lower Pagerank'ed sites have had deeper and fuller indexing, maybe the least internally crosslinked threads have been dropped? Or would Google decide to do less deep crawling on forums?

Then there is Freshness or the frequent spidering and short-indexing of a select number of pages:

Ever since Brett has added a shortlist of the latest participated threads on the member profiles, you can sometimes find Webmasterworld more often when searching in Google for topical subjects, as every once in a while Google decides to refresh-index moderators profiles. (the profile pages of moderators get a new link towards them with every new thread, as the moderator names always are on top of the thread and as far as I am concerned, recent new inbound links is a key criterium for freshness).
Try this for example Greetings from Inktomi [google.nl] . (this example will only last for a short while)

I guess the best remedy for fuller indexing will be Googeguy having a search experience trying to find some of his own threads back again ;)

Added: Then there is also ATW..28.813 listings within webmasterworld.com.. [alltheweb.com]