Welcome to WebmasterWorld Guest from 54.161.116.225

Forum Moderators: phranque

Message Too Old, No Replies

xml sitemap issue

how to create sitemap automatically for huge site

     
6:49 am on Mar 3, 2016 (gmt 0)

New User

joined:Nov 10, 2015
posts: 14
votes: 0


Hello,

I have started working on a website having numbers of folders containing huge numbers of pages. I want to create an index page of sitemap in root to link to all sitemaps in sub-folders.

How can I do it in the least time or what kind of tool I can use for this purpose ? or I should create all sitemaps manually ? Please help
8:47 pm on Mar 10, 2016 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member andy_langton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 27, 2003
posts:3332
votes: 140


Hi Sachet, why do you want to create so many sitemaps? Is there a problem with your site's link structure?
4:30 am on Mar 11, 2016 (gmt 0)

New User

joined:Nov 10, 2015
posts: 14
votes: 0


Andy, I have created a sitemap containing all URLs but it is more than 4 mb in size & URLs are supposed to increase everyday, I want to create small sitemaps for each folder for optimal size.
10:06 am on Mar 11, 2016 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member andy_langton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 27, 2003
posts:3332
votes: 140


Understood, but my question is about whether all your effort to create sitemaps is worth it? You could consider just submitting a sitemap of new pages, for instance. If you're just resubmitting pages that Google is already aware of, this is very unlikely to have any impact on performance.
10:27 am on Mar 11, 2016 (gmt 0)

New User

joined:Nov 10, 2015
posts: 14
votes: 0


I don't think sitemap is only to let search engine know about my new pages, around 20% of total pages of my websites are yet to be indexed & pages are generally updated on time to time. I want all pages to index all pages quickly
11:54 am on Mar 11, 2016 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member andy_langton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 27, 2003
posts:3332
votes: 140


I don't think sitemap is only to let search engine know about my new pages, around 20% of total pages of my websites are yet to be indexed & pages are generally updated on time to time. I want all pages to index all pages quickly


Sitemaps are about forcing Google to discover URLs, for instance:

- When you have a lot of URLs that Google isn't already aware of (e.g. you launch a new site or a large section of new content)
- When you need URLs processed more quickly than may occur naturally
- When you have a lot of changing content

If your content is not being indexed without a sitemap, then you likely have a bigger issue with how your pages are linked together and how many third parties link to you, which is not fixed by a sitemap. Rather, a sitemap will just conceal the underlying problem.
7:10 pm on Mar 11, 2016 (gmt 0)

Preferred Member from GB 

10+ Year Member Top Contributors Of The Month

joined:July 25, 2005
posts: 404
votes: 16


There is a school of thought putting sitemaps on the shelf of "overrated SEO gimmicks"

I must admit I still do them by virtue of habit but I'm yet to be convinced that having a sitemap can improve the crawl stats

If it's a brand-new site, and judging by the OP it is, I'd agree with Andy Langton and only do a sitemap listing the key pages and the fresh pages.

Throwing 4MB of XML at Google won't speed it up. Make sure that each folder and page is linked to in a relevant and logical way and that visitors are discovering your content (not only your home page) via social media and natural links.

In all honesty, I see those full sitemaps as a helping hand to scrapers and content spinners.

Ok, something tells me this is not convincing :) If you want to go ahead with it, I hope you have some sort of a database behind it. The easiest way is to generate sitemaps dynamically from your database. There are plenty of free/cheap scripts out there, just search for "sql sitemap script" or similar...
9:03 pm on Mar 11, 2016 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member andy_langton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 27, 2003
posts:3332
votes: 140


"overrated SEO gimmicks"


I don't know if I would go quite so far, and for some sites sitemaps are extremely useful. But for a typical site, is generating sitemaps the most useful activity? I think this is unlikely.
10:20 pm on Mar 11, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:8684
votes: 693


Do remember that g does have limitations on the size of sitemaps:

50,000 urls per map
<50mb uncompressed per map

Break large sites into smaller sitemaps and use a master sitemap to submit those. In that regard you can have up to 1,000 sitemaps, thus 50,000,000 urls - 1,000
3:29 pm on Mar 12, 2016 (gmt 0)

New User

joined:Nov 10, 2015
posts: 14
votes: 0


If we need to have sitemap with main pages only & rest would be taken cared by website’s internal link structure, then why Google (google.com/sitemap.xml) is using the sitemap structure as Tangor has described above ?
5:33 pm on Mar 12, 2016 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member andy_langton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 27, 2003
posts:3332
votes: 140


why Google (google.com/sitemap.xml) is using the sitemap structure as Tangor has described above ?


Because Google is one of the main proponents of sitemaps. They want to discover as much content as possible. They don't care about where the discovered content ranks.

Note that Google do not perform at all well in search results. They don't rank in the top three for "search engine" for instance, so I would not suggest using the Google website as a model for your own SEO efforts.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members