phpBB indexing?

Google loves it, Yahoo doesn't seem interested ...

[mmarlor]

GoogleBot is very fond of indexing my site's forums - absolutely loves it, and it draws in quite a number of new participants through web searches.

Yahoo, however, has started picking up plenty of articles from my site, and it's even indexed the forum's index page - but it just doesn't seem interested in any topics.

I have previously implemented the phpBB sessions optimization for GoogleBot and Slurp, to remove session ids from URLs. Since Yahoo went live with their engine, I have reduced the UserAgent string search down to just 'slurp', which using strstr in php should find the Yahoo crawler quite fine.

Is there anything I've missed? Or does Yahoo have a thing against phpBB forum pages? :-S

Thanks in advance,

Matt

[roddy]

I've just checked, and Yahoo has 191 of my forum pages, opposed to 30,000 in Google. I haven't made any specific adjustments for search engines, it's pretty much a standard 2.0.6 phpBB package.

Slurp does seem to keep coming back for the index page, which is a PR6 (not that Slurp know that, of course. Or does he?) but not much deeper. The pages that have been picked up don't seem to follow any pattern - ie, they're not ones that have been linked from other sites or anything - I think they're just ones that had 'last post' links when they were spidering.

Roddy

[mikeD]

I have trouble mmarlor getting my phpbb forum indexed. Is there something i need to alter to get it crawled and indexed. The index.php page has a pr6

[mmarlor]

Google is pretty smart and seems to handle session ids anyway. But just in case (way back when I started the site), I applied the phpBB mod to remove session ids for crawlers (in the knowledge base at phpBB, look for article id 29). I've updated the [e-mail] string to just read slurp, not that it's done any good as yet.

In terms of getting the forums crawled effectively, though, I think what Google relies on more is some custom code I created to permit forum topics to be viewed by month from my main site. I created this for the benefit of users, but effectively Google (and other crawlers) have all my forum topics at their finger tips.

My problem here is that even with no session id's - both through the sessions mod mentioned above, and through my "forums by month" functionality, Yahoo more or less ignores the existence of the forums. Very strange indeed.