Welcome to WebmasterWorld Guest from 54.145.58.37

Forum Moderators: rogerd

Message Too Old, No Replies

Getting Google to do a deeper crawl?

     
9:45 am on Dec 15, 2007 (gmt 0)

New User

5+ Year Member

joined:Dec 15, 2007
posts: 3
votes: 0


Good morning ladies and gents. Just had a quick question that I was hoping to tap your creative minds.

I have a forum with about 500 threads. The forum is 6 months old. Google only seems to have indexed about 120 of the 500 topics. Is there any way I can get it to index more?

Everytime I look at googlebot online it says it is viewing profile and viewing private meaasges loads of the time. What the heck is it doing wasting time on this?

Many thanks for any replies.

12:48 pm on Dec 15, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2003
posts: 105
votes: 0


Add a forum sitemap with all thread URL;s and submit it to Google.
8:54 pm on Dec 15, 2007 (gmt 0)

New User

5+ Year Member

joined:Dec 15, 2007
posts: 3
votes: 0


Thanks Ning, any online guides showing me how to do that?

Thanks

8:54 am on Dec 17, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 4, 2004
posts:877
votes: 0


Forums, at least my experience with phpbb, are a minefield of duplicate content and uneeded content. To give you an example I had a forum with soemthing 12,000 pages indexed where the actual count of real content excluding profiles and the like was probably more like 1000. The bots simply get overwhelmed with and have no idea where to go..

If it's viewing PM's and profiles then you must not have denied it access to those pages with robots.txt That's the first place to start.

You may also note that if you're using phpbb2 or 3 that is only the tip of the iceberg, phpbb2 has upwards of 10 URL's per actual page just through pagination and other features. If you let it into the search page you're looking at some ridiculous amount.

phpBB3 is a little better and hides some content from bots but still has about 5 duplicate URL's per page.

9:16 am on Dec 17, 2007 (gmt 0)

New User

5+ Year Member

joined:Dec 15, 2007
posts: 3
votes: 0


Thanks coalman do I need to put this into the robot file?

User-Agent: *
Disallow: /profile.php/
/private messages.php/

Thanks

10:05 am on Dec 17, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 7, 2004
posts:929
votes: 0


Yep, you can use the robot.txt file to restrict access to some URL.

Thanks Ning, any online guides showing me how to do that?

Just try the Google help from Google Webmaster Tools, there is a long explanation on how to create sitemap in XML.
12:42 am on Dec 18, 2007 (gmt 0)

Administrator

WebmasterWorld Administrator rogerd is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 2, 2000
posts:9686
votes: 0


If you want to stimulate deep crawling by Google, try to improve external linkage. Good links to your forum home page will help, as will some links to individual forums and threads.
12:51 am on Dec 18, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 19, 2006
posts:61
votes: 0


Strong pagerank for the main domain, simplified navigation, and sitemaps work well for Google. Have yet to figure out how to get Yahoo and MSN (even with sitemaps) to deep crawl.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members