homepage Welcome to WebmasterWorld Guest from 54.145.172.149
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Community Building and User Generated Content
Forum Library, Charter, Moderators: rogerd

Community Building and User Generated Content Forum

    
Getting Google to do a deeper crawl?
jimcanswim

5+ Year Member



 
Msg#: 3528422 posted 9:45 am on Dec 15, 2007 (gmt 0)

Good morning ladies and gents. Just had a quick question that I was hoping to tap your creative minds.

I have a forum with about 500 threads. The forum is 6 months old. Google only seems to have indexed about 120 of the 500 topics. Is there any way I can get it to index more?

Everytime I look at googlebot online it says it is viewing profile and viewing private meaasges loads of the time. What the heck is it doing wasting time on this?

Many thanks for any replies.

 

ning

10+ Year Member



 
Msg#: 3528422 posted 12:48 pm on Dec 15, 2007 (gmt 0)

Add a forum sitemap with all thread URL;s and submit it to Google.

jimcanswim

5+ Year Member



 
Msg#: 3528422 posted 8:54 pm on Dec 15, 2007 (gmt 0)

Thanks Ning, any online guides showing me how to do that?

Thanks

thecoalman

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3528422 posted 8:54 am on Dec 17, 2007 (gmt 0)

Forums, at least my experience with phpbb, are a minefield of duplicate content and uneeded content. To give you an example I had a forum with soemthing 12,000 pages indexed where the actual count of real content excluding profiles and the like was probably more like 1000. The bots simply get overwhelmed with and have no idea where to go..

If it's viewing PM's and profiles then you must not have denied it access to those pages with robots.txt That's the first place to start.

You may also note that if you're using phpbb2 or 3 that is only the tip of the iceberg, phpbb2 has upwards of 10 URL's per actual page just through pagination and other features. If you let it into the search page you're looking at some ridiculous amount.

phpBB3 is a little better and hides some content from bots but still has about 5 duplicate URL's per page.

jimcanswim

5+ Year Member



 
Msg#: 3528422 posted 9:16 am on Dec 17, 2007 (gmt 0)

Thanks coalman do I need to put this into the robot file?

User-Agent: *
Disallow: /profile.php/
/private messages.php/

Thanks

tomda

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3528422 posted 10:05 am on Dec 17, 2007 (gmt 0)

Yep, you can use the robot.txt file to restrict access to some URL.

Thanks Ning, any online guides showing me how to do that?

Just try the Google help from Google Webmaster Tools, there is a long explanation on how to create sitemap in XML.

rogerd

WebmasterWorld Administrator rogerd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3528422 posted 12:42 am on Dec 18, 2007 (gmt 0)

If you want to stimulate deep crawling by Google, try to improve external linkage. Good links to your forum home page will help, as will some links to individual forums and threads.

JonW

5+ Year Member



 
Msg#: 3528422 posted 12:51 am on Dec 18, 2007 (gmt 0)

Strong pagerank for the main domain, simplified navigation, and sitemaps work well for Google. Have yet to figure out how to get Yahoo and MSN (even with sitemaps) to deep crawl.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Community Building and User Generated Content
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved