Getting Google to index my discussion pages

[Nitin]

I want my discussion pages to be indexed by Googlebot.My discussion pages are generated when a cgi script is executed.All my discussion pages are in cgi-bin directory which is not accessible to public and all pages are too deep buried.

How deep does Googlebot crawl our webpages.My discussion pages contain very valuable information and i want them to be indexed.

Thanks for your time.

[BGumble]

Hello Nitin, welcome to Webmasterworld!

Can you tell us the name of the software your forum uses? That will help narrow down the possibilities for an archive. You have three choices:

-Enable Google to read the actual posts
or
-Write a new interface for search engines to read the actual posts
or
-Write a script to dump all of the CGI database information out to flat HTML that engines can read

Hopefully one of those options will already have been done for your forum software.

[Nitin]

Thanks BGumble,

I am using Ultimate Bulletin Board.

How can I enable Google to read the actual post.In my robots.txt i have written

User-agent: *
Disallow:

[BGumble]

UBBThreads or UBBClassic?

I believe UBBClassic writes out HTML that works fine so I'll assume you have UBBThreads. I've stickied you with two URLs-- the first is a thread from the UBB forum the second is a commercial application that will specifically get your UBB spidered.

[beachlover]

Actually I have the same problem with my forum. It is also cgi generated (the software is YaBB) and Google denies to crawl it.

Is there an easy way to make the bot crawl the pages?

[dotremy]

Interesting, I will drop my forum as well, its a Ikonboard.cgi

If I could get the bot to follow it I would have 40.000 valuable content pages extra.

[Yidaki]

>ultimeate bb, yabb, iconboard ...
>40.000 valuable content pages

Time to read about url rewriting using either the mod_rewrite module apache has to offer, or contact the support forums of the software you're using. Many out of the box bb forums have their own rewrite engines / codes.

Use the site search [webmasterworld.com] here at WebmasterWorld ... there have been some interesting and helpfull threads about this topic in the past.

Hint: search google's own WebmasterWorld site search ;) for the topic rewrite dynamic url's to static url's [google.de].

[BGumble]

The problem with Ikonboard is the session ID in the query string. Before version 3, Ikonboards would be spidered without a problem. I couldn't find any information where someone has already done this, so best would be to post on the Ikonboard support and tell them you'd like to remove sessionIDs from the query string-- you can do it only for Google by identifying the User Agent.

YaBB has some ugly URLs. I've stickied you with a topic someone started this morning about this very subject.