Welcome to WebmasterWorld Guest from 54.162.157.249

Forum Moderators: rogerd

Message Too Old, No Replies

Getting Google to do a deeper crawl?

     
9:45 am on Dec 15, 2007 (gmt 0)

5+ Year Member



Good morning ladies and gents. Just had a quick question that I was hoping to tap your creative minds.

I have a forum with about 500 threads. The forum is 6 months old. Google only seems to have indexed about 120 of the 500 topics. Is there any way I can get it to index more?

Everytime I look at googlebot online it says it is viewing profile and viewing private meaasges loads of the time. What the heck is it doing wasting time on this?

Many thanks for any replies.

12:48 pm on Dec 15, 2007 (gmt 0)

10+ Year Member



Add a forum sitemap with all thread URL;s and submit it to Google.
8:54 pm on Dec 15, 2007 (gmt 0)

5+ Year Member



Thanks Ning, any online guides showing me how to do that?

Thanks

8:54 am on Dec 17, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Forums, at least my experience with phpbb, are a minefield of duplicate content and uneeded content. To give you an example I had a forum with soemthing 12,000 pages indexed where the actual count of real content excluding profiles and the like was probably more like 1000. The bots simply get overwhelmed with and have no idea where to go..

If it's viewing PM's and profiles then you must not have denied it access to those pages with robots.txt That's the first place to start.

You may also note that if you're using phpbb2 or 3 that is only the tip of the iceberg, phpbb2 has upwards of 10 URL's per actual page just through pagination and other features. If you let it into the search page you're looking at some ridiculous amount.

phpBB3 is a little better and hides some content from bots but still has about 5 duplicate URL's per page.

9:16 am on Dec 17, 2007 (gmt 0)

5+ Year Member



Thanks coalman do I need to put this into the robot file?

User-Agent: *
Disallow: /profile.php/
/private messages.php/

Thanks

10:05 am on Dec 17, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yep, you can use the robot.txt file to restrict access to some URL.

Thanks Ning, any online guides showing me how to do that?

Just try the Google help from Google Webmaster Tools, there is a long explanation on how to create sitemap in XML.
12:42 am on Dec 18, 2007 (gmt 0)

WebmasterWorld Administrator rogerd is a WebmasterWorld Top Contributor of All Time 10+ Year Member



If you want to stimulate deep crawling by Google, try to improve external linkage. Good links to your forum home page will help, as will some links to individual forums and threads.
12:51 am on Dec 18, 2007 (gmt 0)

5+ Year Member



Strong pagerank for the main domain, simplified navigation, and sitemaps work well for Google. Have yet to figure out how to get Yahoo and MSN (even with sitemaps) to deep crawl.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month