Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Sitemaps Generator on Large Site

Could this cause problems?

4:57 pm on Feb 23, 2009 (gmt 0)

5+ Year Member

Sorry if this sounds like a newbie question. Could it be possibly harmful to run a sitemap generator on a large site? I am trying one now that seems to send out six different crawlers, and renders a pageview for each visit (which normal bots don't usually do). Can this cause a problem on my server? (overload of something, waste of resources, etc.)


3:36 pm on Feb 24, 2009 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

hi lance,

running a bad piece of any software could be harmful. i assume your sitemap generator was not custom built by a former ddos attacker. if you use any of the common sitemap generators you should be fine.

your server should be able to deal with creating the sitemap. it may be wasting some resources but i am guessing it is probably not significant waste. if you are scheduling this to run every 24 hours (which is probably overkill) you should schedule it to run during your lowest traffic hour (generally middle of the night).

the bigger issue might be your reporting getting distorted. many analytic programs allow you to remove your own ip from reporting. that should remove these extra page views from reporting.

4:31 am on Feb 27, 2009 (gmt 0)

5+ Year Member

yes, the problem is adsense seems to be tracking pageviews, so I'm having to disable all ads before running. I will definitely not be running it daily though. Thanks for the input on the server.


5:19 pm on Feb 27, 2009 (gmt 0)

5+ Year Member

The reason being is Adsense is showing stats for impressions I would rather not have.
6:23 pm on Feb 27, 2009 (gmt 0)

5+ Year Member

Although the drawback of having to disable adsense and stats (unless anyone knows a way around this), is I am seeing a lot about my site. I'm monitoring it as it is going, even seeing bad links and what not. So all in all, definitly worth it. I may plan on doing it section by section. Thanks for all your help.


[edited by: CWebguy at 6:24 pm (utc) on Feb. 27, 2009]


Featured Threads

Hot Threads This Week

Hot Threads This Month