Msg#: 3855918 posted 4:57 pm on Feb 23, 2009 (gmt 0)
Sorry if this sounds like a newbie question. Could it be possibly harmful to run a sitemap generator on a large site? I am trying one now that seems to send out six different crawlers, and renders a pageview for each visit (which normal bots don't usually do). Can this cause a problem on my server? (overload of something, waste of resources, etc.)
Msg#: 3855918 posted 3:36 pm on Feb 24, 2009 (gmt 0)
running a bad piece of any software could be harmful. i assume your sitemap generator was not custom built by a former ddos attacker. if you use any of the common sitemap generators you should be fine.
your server should be able to deal with creating the sitemap. it may be wasting some resources but i am guessing it is probably not significant waste. if you are scheduling this to run every 24 hours (which is probably overkill) you should schedule it to run during your lowest traffic hour (generally middle of the night).
the bigger issue might be your reporting getting distorted. many analytic programs allow you to remove your own ip from reporting. that should remove these extra page views from reporting.
Msg#: 3855918 posted 6:23 pm on Feb 27, 2009 (gmt 0)
Although the drawback of having to disable adsense and stats (unless anyone knows a way around this), is I am seeing a lot about my site. I'm monitoring it as it is going, even seeing bad links and what not. So all in all, definitly worth it. I may plan on doing it section by section. Thanks for all your help.
[edited by: CWebguy at 6:24 pm (utc) on Feb. 27, 2009]