Search Engines and Databases

Can spiders find my forum pages?

[paul_sully]

Hi,
Thanks for the info,

I have a php forum on my website (phpbb2), can google search page contents even though they are in a forum database?

Paul

[Giacomo]

I have a php forum on my website (phpbb2), can google search page contents even though they are in a forum database?

Googlebot does follow dynamic URLs (i.e., URLs with a querystring such as

?forumid=1&threadid=2

in them). However, there are a few limitations: first of all, Googlebot limits the number of dynamic pages it will crawl; secondly, as repeatedly stated by GoogleGuy, crawling URLs with a PHP session ID in them can be a real pain in the a$$ for Googlebot, so you'd better strip those off your forum URLs if the UserAgent is

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

.

Since this has all been discussed before, I suggest you do a site search of

google dynamic pages

or

google dynamic urls

or

google querystring

on WebmasterWorld. Alternatively, you may use Google to search this site specifying

site:webmasterworld.com

along with your keywords.

Then, once again, make sure you read and understand Google's Information for Webmasters [google.com]: the answers to most recurring questions are right there, in the guidelines [google.com] or in the FAQs [google.com].

About your specific question, yes, there is a way to do that (i.e., make your forum URLs static-looking and strip off the PHP session ID if UserAgent == Googlebot) with phpBB. However, to make phpBB spider-friendly is not a simple task and requires hand-modifying the phpBB code following the instructions in a couple of phpBB "hacks". Unfortunately, AFAIK those hacks have not been packed together and released as a single MOD ("official" phpBB modification) yet, so you will have to go through several posts and "merge" the instructions you will find in there.

A few starting points from the phpBB support forum:

[Tutorial] Google & phpBB [phpbb.com]
Search Engine Friendly and Customer Header / Footer [phpbb.com]
[Reason] How Google spiders PHPBB! [phpbb.com]
Unusual Request [phpbb.com]

Following the instructions from the above posts I have been able to obtain a modified phpBB which is totally spider-friendly. So it is not impossible to do. It only takes some patience.

Good Luck! ;)

[edited by: WebGuerrilla at 7:50 pm (utc) on Mar. 12, 2003]
[edit reason] forum move [/edit]

[paul_sully]

Is it right that googlebot can follow src links? like framesets?

Can it follow javascript menu's that dont use href?

PAul

[Giacomo]

Paul,

Before posting a new question, please take some time to scan through previous posts: your question might have been already answered.

Also, please start a new thread whenever posting a new, unrelated question.

> Is it right that googlebot can follow src links? like framesets?

Yes it is.

> Can it follow javascript menu's that dont use href?

Currently, no.