Forum Moderators: Robert Charlton & goodroi
In a nutshell: I think it's pretty safe to assume that you want your best URLs in the site's XML sitemap. By "best" I mean all angles - interesting content, useful for visitors, has a chance to rank for something, at least long tail. And, of course has an ad revenue potential - as a practicing webmaster I cannot loose sight of that angle, too.
I am using pagination on my forum site: 5 posts in the same thread make up a page. Post #6 of that thread would be on page #2 and so on. Pages of the thread are all interlinked.
For the lack of a better tactic right now I only include Page #1 of each thread in the XML sitemap. Comes up to something like 42,000 URLs in the sitemap (an old, well traveled site). Needless to say, the actual number of pages is above 100,000 because many threads have multiple pages.
So, more often than not I see that by the time the discussion gets under way and solutions are offered (and, honestly, good keywords start to be used, too) and basically the thread becomes interesting, it's already on page #2 or beyond. Many threads start just like this one: a noob asks a question in the most generic terms, then more experienced users ask him/her a few questions to find out exactly what may be the issue and before you know it, page #1 is over before the discussion has even warmed up.
Some pages other than #1 are indexed by G and do show up in results and some (very few) even rank. But most often, if a page ranks for anything, it's the page #1 even though pages 2 and up may have even better content.
So, what does the collective wisdom of this respectful group say about including *ALL* the pages in the XML sitmap. Is is safe (URL is almost the same except for the page #)? Does it even matter due to the fact that page #1 has more incoming links (internal navigation)? Is it wise to increase the size of the XML sitemap so much risking more pages will go un-indexed?
Any comment on the matter will be greatly appreciated!
Thanks!
Thanks, Ted. I do realize XML sitemap is not the only way Google knows about URLs on the site. However, it appears that in this case Gbot really needs some help discovering/visiting the URLs other than page 1. So my line of thought was that XML could be that additional help. So, I guess paraphrasing the original question I would ask:" will adding pages 2,3 etc. to the XML sitemap rather help, hurt or make no difference in those pages' positions(ranks) in Google search?"
So the answer to your question above is: Directly, it should not make them rank better or worse. Indirectly, helping them locate the URLs should probably be seen as a benefit, since if they are not found they cannot rank, but in reality, the pages should show at the same place in the results (when found and indexed) whether you choose to use an XML sitemap or not, and as tedster said: It certainly isn't going to hurt.
A couple more thoughts - how's the forum's link structure? Are the deeper pages only inside a narrow "click silo", or are there other links pointing to them as well? And have you eliminated multiple url problems that create duplicate issues? That would save googlebot some crawling cycles and result in a deeper and more frequent crawl of the unique urls.
Regarding different titles:
Currently the titles are only slightly different as in
page 1: "What a great topic this is"
page 2: "What a great topic this is - Page 2"
and so on. Frankly, even though I can devise a way to title them differently (though I'm out of ideas about what the other titles should/would be based upon) but I am afraid that anything I do will be seen as bending the rules for SERP sake. After all, all pages belong to the same thread and are about the same topic.
Link structure:
I guess you can call it a "link silo" - before you encounter a link to Page 2 (or any other page for that matter) you have to open Page 1. Page 1 is linked to from category page and also, if it's fresh enough, it's linked from the homepage for awhile. Do I hear that this link structure also does not help my case?
That's correct - once you open page 1 or any other page for that matter, you'll have a tree-like navigation section where all other pages are linked from. So, all pages of a thread are thoroughly interlinked but only page 1 has a link from the "outside".
I could throw some links to pages 2, 3 etc from the category page but the only way to do it in the space that's available would be to use the the numbers as the anchor text. Page 1 is linked to using the title as the anchor text.
Do you think that adding this rather bare link (no keyword in the anchor whatsoever) is better than not having a link at all?
And as far as the XML Sitemap is concerned - I'm going to add all pages in addition to #1 that's already there. It seems that the common wisdom is that it will at least not hurt.
And yes, a link without anchor text still votes PR to the target url. You might also consider hand picking some real winner deep pages to link to from time to time -- and use anchor text in that case, since you'd already be giving it manual attention.
EG:
Your Page Title => 2, 3, 4
Just my .02.
<added>
One other thought I have, and like tedster says it should be tested on a small sample, is to double the posts-per-page, like a very popular forum we know and frequent did... The minimum is now 20. I can't even set it back to 10 anymore.
Have I complained about that yet, Brett? I keep thinking I need to send you a scathing sticky... Give me my 10 posts-per-page back!
</added>
Thanks for your advice, guys. All makes sense. I will try the pagination TheMadScientist suggested, give it a couple of months (enough?) and then do the other pages in XML sitemap in case one of the changes will need to get rolled back.
