homepage Welcome to WebmasterWorld Guest from 54.227.34.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Sitemaps Generator on Large Site
Could this cause problems?
CWebguy

5+ Year Member



 
Msg#: 3855918 posted 4:57 pm on Feb 23, 2009 (gmt 0)

Sorry if this sounds like a newbie question. Could it be possibly harmful to run a sitemap generator on a large site? I am trying one now that seems to send out six different crawlers, and renders a pageview for each visit (which normal bots don't usually do). Can this cause a problem on my server? (overload of something, waste of resources, etc.)

Thanks!
Lance

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3855918 posted 3:36 pm on Feb 24, 2009 (gmt 0)

hi lance,

running a bad piece of any software could be harmful. i assume your sitemap generator was not custom built by a former ddos attacker. if you use any of the common sitemap generators you should be fine.

your server should be able to deal with creating the sitemap. it may be wasting some resources but i am guessing it is probably not significant waste. if you are scheduling this to run every 24 hours (which is probably overkill) you should schedule it to run during your lowest traffic hour (generally middle of the night).

the bigger issue might be your reporting getting distorted. many analytic programs allow you to remove your own ip from reporting. that should remove these extra page views from reporting.

CWebguy

5+ Year Member



 
Msg#: 3855918 posted 4:31 am on Feb 27, 2009 (gmt 0)

yes, the problem is adsense seems to be tracking pageviews, so I'm having to disable all ads before running. I will definitely not be running it daily though. Thanks for the input on the server.

Lance

CWebguy

5+ Year Member



 
Msg#: 3855918 posted 5:19 pm on Feb 27, 2009 (gmt 0)

The reason being is Adsense is showing stats for impressions I would rather not have.

CWebguy

5+ Year Member



 
Msg#: 3855918 posted 6:23 pm on Feb 27, 2009 (gmt 0)

Although the drawback of having to disable adsense and stats (unless anyone knows a way around this), is I am seeing a lot about my site. I'm monitoring it as it is going, even seeing bad links and what not. So all in all, definitly worth it. I may plan on doing it section by section. Thanks for all your help.

Lance

[edited by: CWebguy at 6:24 pm (utc) on Feb. 27, 2009]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved