Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Advice on sitemap for rapidly changing site?

         

punisa

1:15 am on Jun 5, 2009 (gmt 0)

10+ Year Member



Hi ! I've created a sitemap for my site a year ago. Google webmaster tools acknowledged its existence.
Haven't actually changed it since then.

Now my site changed, it became a very dynamic sort of web 2.0 site.
I have approximately 1000 new pages added each day.
What should I do about sitemap?
Should I just delete it or?

I couldn't possibly add all relevant links to it : (

And Google crawls our site great, almost all articles end up high in results.

What would happen if I abandon sitemap? Would Google change its attitude towards my site?

tedster

1:56 am on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



A sitemap is supplemental information, added to Google's normal crawling and indexing. You can drop it without having that action create a "black mark" on your record.

However, it sounds like you are now in a situation where a sitemap might be a better thing to implement than it used to be. You can automate the sitempa generation process on your server, creating and submitting a new sitemap automatically up to once per hour. I'd look into doing that.

Here's one reference from Google about a recurring sitemap script:
[google.com...]

punisa

1:53 pm on Jun 5, 2009 (gmt 0)

10+ Year Member



Thanks for clearing that out Tedster : )
I like to double check everything when it comes to Google : D

I already have some scripts running in a sort of automated mode, I believe I could create a creation of sitemap along with other actions.

Just a few more questions:
1. should I include all links in my sitemaps? This might grow very large very fast. Will it choke bots? For example, what happens if your sitemap file is over 1 MB?
Should I in such case just "show" bots the way to the main categories and keyword tag sections? (my categories are constructed something as www.example.com/category/tag/)

2. I'm not very knowledgable with scripting options google reccomends for creating a dynamic sitemap, but I could easily do it with PHP. Where could I find a simple outline and what actually needs to go inside sitemap.xml file?
Should I include only basic links or additional parameters? I saw somewhere that sitemaps can include stuff like crawl rate etc..

punisa

3:03 pm on Jun 5, 2009 (gmt 0)

10+ Year Member



I did a quick draft of my to-be sitemap and here are the stats -
ACTUAL numbers:
- current size approx. 1 MB
- number of current links approx 9.200
- average link increase per 30 days - 13.000

From these numbers we can see that my sitemap will actually grow by 1 MB every month.
I'm not sure how will Google like this.. What is the procedure in such occasions?
I've read some articles at Google but didn't find this topic addressed.

Do I need to break up my sitemap into pieces or?

tedster

7:58 pm on Jun 5, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you want to list more than 50,000 URLs, you must create multiple Sitemap files. If you anticipate your Sitemap growing beyond 50,000 URLs or 10MB, you should consider creating multiple Sitemap files. If you do provide multiple Sitemaps, you can list them in a Sitemap index file. Sitemap index files may not list more than 1,000 Sitemaps.

[google.com...]