Sitemaps Generator on Large Site

Could this cause problems?

[CWebguy]

Sorry if this sounds like a newbie question. Could it be possibly harmful to run a sitemap generator on a large site? I am trying one now that seems to send out six different crawlers, and renders a pageview for each visit (which normal bots don't usually do). Can this cause a problem on my server? (overload of something, waste of resources, etc.)

Thanks!
Lance

[goodroi]

hi lance,

running a bad piece of any software could be harmful. i assume your sitemap generator was not custom built by a former ddos attacker. if you use any of the common sitemap generators you should be fine.

your server should be able to deal with creating the sitemap. it may be wasting some resources but i am guessing it is probably not significant waste. if you are scheduling this to run every 24 hours (which is probably overkill) you should schedule it to run during your lowest traffic hour (generally middle of the night).

the bigger issue might be your reporting getting distorted. many analytic programs allow you to remove your own ip from reporting. that should remove these extra page views from reporting.

[CWebguy]

yes, the problem is adsense seems to be tracking pageviews, so I'm having to disable all ads before running. I will definitely not be running it daily though. Thanks for the input on the server.

Lance

[CWebguy]

The reason being is Adsense is showing stats for impressions I would rather not have.

[CWebguy]

Although the drawback of having to disable adsense and stats (unless anyone knows a way around this), is I am seeing a lot about my site. I'm monitoring it as it is going, even seeing bad links and what not. So all in all, definitly worth it. I may plan on doing it section by section. Thanks for all your help.

Lance

[edited by: CWebguy at 6:24 pm (utc) on Feb. 27, 2009]