homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Generating a huge sitemap on shared host
Creating a sitemap with thousands of URL'son a shared hosting plan.

Msg#: 4316408 posted 5:54 pm on May 23, 2011 (gmt 0)

I'm looking into creating a sitemap for my site running joomla and phpbb3. I've encountered a couple problems and have a few questions on what's really needed and how to get around these problems. The site has thousands of pages because of the message board. This is the "meat and potatoes" of the site and my goal as the webmaster is to grow the user base so that I get more enjoyment out of the site and maybe make enough coin to pay the bills.

Question 1 - Is there a noticeable benefit to having a sitemap containg URL's for every relevant topic compared to a sitemap with URL's to the main pages? Would manually creating a sitemap that goes a view levels deep into the forum be sufficient?

I ask this because I am having issues generating a sitemap. I tried using gsitecrawler and, because of the size, it puts a strain on my host and takes a LONG time to run. I am on a shared hosting plan and got scolded for going above "industry standard usage levels" last time I ran the crawler. I was able to explain that the usage was an anomaly and they granted me a window to get the sitemap created. This leads me to question 2:

Question 2 - If I use gsitecrawler, how often will I need to run it and produce an up to date sitemap? I think the most I could weasel out of my host is once a month, would this be sufficient?

I also looked into using googles python script. (http://sitemap-generators.googlecode.com/svn/trunk/docs/en/sitemap-generator.html) I'm not even sure this would work with my message board as it uses " create Sitemaps from URL lists, web server directories, or from access logs." Plus, my limited host comes into play again as they want a $40 per year fee to be able to access via SSH. Also, I'm thinking this would create usage issues as well, yes?

Question 3 - Is the google python script a viable alternative? Is there any other method you can suggest to accomplish my goal considering my limitations?

As always, thanks for your help!


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved