Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: martinibuster
Yahoo, however, has started picking up plenty of articles from my site, and it's even indexed the forum's index page - but it just doesn't seem interested in any topics.
I have previously implemented the phpBB sessions optimization for GoogleBot and Slurp, to remove session ids from URLs. Since Yahoo went live with their engine, I have reduced the UserAgent string search down to just 'slurp', which using strstr in php should find the Yahoo crawler quite fine.
Is there anything I've missed? Or does Yahoo have a thing against phpBB forum pages? :-S
Thanks in advance,
Slurp does seem to keep coming back for the index page, which is a PR6 (not that Slurp know that, of course. Or does he?) but not much deeper. The pages that have been picked up don't seem to follow any pattern - ie, they're not ones that have been linked from other sites or anything - I think they're just ones that had 'last post' links when they were spidering.
In terms of getting the forums crawled effectively, though, I think what Google relies on more is some custom code I created to permit forum topics to be viewed by month from my main site. I created this for the benefit of users, but effectively Google (and other crawlers) have all my forum topics at their finger tips.
My problem here is that even with no session id's - both through the sessions mod mentioned above, and through my "forums by month" functionality, Yahoo more or less ignores the existence of the forums. Very strange indeed.