Possible downside of removing sitemap

[atlrus]

I am adding a forum to my website and obviously I won't be able to keep up with the sitemap. And I was wondering - should I remove the sitemap from WMT or keep it (incomplete)?

Anyone experienced a downside of removing the sitemap from Google's WMT?

[goodroi]

I have rarely seen adding or removing sitemaps to dramatically impact a website. Just make sure your internal link structure is good and the site has no big technical issues.

You may want to keep the sitemap as a way to gather intelligence on which pages Google has indexed via Webmaster Central.

[Sgt_Kickaxe]

I removed the sitemap on my best site nearly a year ago and did not suffer ANY noticeable ranking loss. The site, being older and well established, is crawled regularly and the sitemap was not providing any real benefit imo.

By removing it I have realized a performance benefit since it is no longer drawing resources from my database.

I can't say that I noticed any change in Bing/Yahoo either though I didn't watch them as closely. I gather stats from other sources so I didn't need the information provided by GWT but it is useful if you have to internal tracking.

[robert76]

I have rarely seen adding or removing sitemaps to dramatically impact a website.

On a related topic, never had a sitemap in our WMT. Very old ecomm site that lost over 50% of traffic in Panda 1.0 and has not recovered.

Does this particular scenario change anything? Would adding at this point in time benefit us or hurt us more?

[Marshall]

According to Google, they recommend you use a site map to help them find pages they might not otherwise. As long as all of your pages are properly linked, there should not be an issue. I know I have some sites with maps, and some without, and there is no obvious difference.

Marshall

[lucy24]

It doesn't matter if there are pages on your site that aren't on the sitemap, so long as they're linked. Search engines will find them. What does matter is when there are pages on the sitemap that the search engine can't find. (Bing in particular says some ominous things about "not trusting" your sitemap if it's got too many 301s.)

For a forum, you may actually be better off not having a sitemap. It isn't just adding material. You have to keep track of every thread that is moved/ merged/ deleted and adjust the sitemap accordingly. G### will still raise a ruckus every time it can't find something that was there in 2007, but at least they can't claim it's your fault.

[sociomedia21]

Maybe I'm wrong but from your introduction I understood that you don't want to continue with the sitemaps mechanic because by adding a forum it may become a total mess.

If I'm correct, I would like to add that you can use different sitemap files for forum and the rest of the Web, and submit them individually, or create a master sitemap file that links to the other ones and submit the master as the WP SEO pluggin from yoast does.

This way, you can keep the current sitemap for the regular site, and create a second one totally independant dfor the forum via some plugin if it's available.

[lucy24]

The problem is that Google never forgets a sitemap. Sometimes they'll list a file as Not Found, and in the "our feeble excuse for looking for this imaginary file" column it will say something like On Sitemap From October 2010. Never mind that it isn't on the current sitemap, or the one before, or the one before...

:: vague mental association with Roach Motel here ::

[keyplyr]

Keep your sitemap.xml and just add the forum as a subdomain (with respective A record change) Example: forum.domain.com.

Then add a separate sitemap.xml for that subdomain which could be tagged for hourly changefreq even though you would not be updating the time stamp.

Not quite perfect, but close. Google just uses sitemap.xml as a suggestion anyway.

[zehrila]

My main site doesn't have sitemaps, it has some url related issues, for which i did not submit it. Traffic is increasing every day.

[atlrus]

Did not realize this thread made the front page.

A couple of days after my first post I decided to remove the sitemap from GWT. But I did not remove the sitemap itself from the server.

So far I have not seen anything negative and Google is indexing my site and forums (in a sub-folder rather than subdomain) in a satisfactory matter.

Maybe I'm wrong but from your introduction I understood that you don't want to continue with the sitemaps mechanic because by adding a forum it may become a total mess.

You were correct, there is no way I could keep up with a sitemap including a forum :)

[AjiNIMC]

These are the issues, you might face if you remove sitemap, or if you have a bad sitemap.

If Google becomes depended on your sitemap and then if the sitemap stops working, it can end up not crawling some of the new pages. It happened with me but that was around 2 years back ... now may be better.

Sitemap helps you rank better esp if that is realted to keywords that falls under "QDF". [youtube.com...]

[ZydoSEO]

Personally, I think sitemaps.xml files are EXTREMELY overrated and often misused. A typical web site if well architected with good internal linking structures (top navs, left navs, breadcrumbs, footers, etc.) should never require a sitemap.xml file, even one with thousands of URLs. Most webmasters are simply wasting their time with sitemap.xml files.

IMO there are only a couple of situations where a sitemap.xml is needed:

1) If your site has crawlability issues that would prevent the engines from discovering all of your important URLs during the normal crawl process and
2) If your site has 100s of thousands or millions of URLs.

In scenario 1 above, a sitemap.xml can actually help get URLs indexed that might not otherwise have been discovered during a normal crawl of the site.

In scenario 2 above, a sitemap.xml does let you prioritize which URLs you want the engines to index from your site using the <priority> element. You can basically tell them, "Index this set of URLs first. If you still have room in your index for more of my site's URLs then index this set of URLs next. And if your still have room in your index for more of my site's URLs then index this set of URLs next..." and so on.

Other than those two scenarios, I think they are truly a waste. I think it's funny how people will throw up a brand new site and be so worried about getting a sitemap.xml created for it so that they can submit it and get indexed. Why bother? You might get it indexed by submitting a sitemap.xml, but it's not going to rank without any kind of links. In fact it could be de-indexed a few days or weeks later if it doesn't acquire any inbound links. Getting a new site indexed with a sitemap.xml is pretty much worthless. That time could be better spent building links and waiting on the engines to discover and index your site the old fashion way - by following links.

Google's recommendation that you use them is just a "party line"... kind of like your car dealer telling you that you should get your oil changed every 3750 miles.

Sitemaps do not get your site crawled more often, they don't get more URLs indexed (unless your site has crawlability issues), and they won't help your rankings. Even Google tells you they are used to "assist" in crawling. But they are not even bound to following the priorities you might set in the sitemap.xml. If inbound links exist to one of your lowest priority URLs, they will likely still index that lowest priority URL before a higher priority URL with no inbound links.

So unless your site falls into one of the 2 scenarios above (one with crawlability issues or a huge site), removing your sitemap.xml should not negatively impact your site's rankings.

[rango]

I just added a more complete sitemap to our site, which includes a forum with over 80k threads. Obviously I did not make it manually. I created a script to do it that updates on a daily basis.

The main benefit for me is that through Webmaster tools I can now clearly see the number of urls that I would like to have indexed.

Prior to this we just had a sitemap with our main static urls. I don't think it had any negative effect.

[MikeNoLastName]

If you're talking a "user-navigational" sitemap.htm, which links out to many pages and gets linked TO by many pages on your site for navigational purposes, I just recently removed ours and feel it has improved things a lot. Reason being I realized it was getting a lot of low quality internal backlinks and thus a high PR, but why would anyone ever search for "YourSite Sitemap"? Better to let that PR flow directly to the other closer related pages on your site.

Now if you are talking about a sitemap.xml intended for search engines, I consider it invaluable, particularly if you have a lot of pages being changed, added or updated weekly. I have found that it encourages frequently updated pages to get recrawled a lot quicker, and new pages are found usually within a day or two max. Google has (had?) a routine that you can run manually or have your server OS run periodically which will entirely rebuild your sitemap.xml FOR you in seconds by "Walking" your directory structure according to rules you set, so there is no real manual labor involved in keeping it up to date.

[martinibuster]

Personally, I think sitemaps.xml files are EXTREMELY overrated

I agree. I kind of get the feel that the xml standard was created to lure webmasters into the GWT/Analytics crack, with Bing/Yahoo along as unknowing facilitators. There is more benefit to be had from an html site map, particularly in a forum that might generate duplicate pages caused by dynamically generated url strings.

For a forum, you may actually be better off not having a sitemap. It isn't just adding material. You have to keep track of every thread that is moved/ merged/ deleted and adjust the sitemap accordingly.

There are mods that dynamically generate both/either HTML and XML site maps. Change the forum and the site map changes with it.

there is no way I could keep up with a sitemap including a forum

There is no need to manually keep up with a site map in a forum. As I mentioned before, popular forums have mods that can be applied to dynamically generate a site map of the content. Yes it can be huge but it can also help sort out dupe content issues. Traffic spiked years ago when I added an HTML site map and kept on climbing year over year.

But now I'm interested in hearing what could happen if a site map is removed from any kind of website, particularly a forum.

[travelin cat]

About two years ago we removed the sitemap on our site of approx. 11,000 pages, 10,000 of which (approx.) where forum posts. Never saw a hit/change to any of our traffic.

I was reluctant to remove it, but it had become unwieldily. So we did a bit of research that pointed to the fact that there really is no need to have one if of you have good site architecture combined with thorough interlinking.

[keyplyr]

While having good site link architecture is indeed important, having an updated sitemap.xml alerts the SE which pages have changed and when. That's the point.

[DirigoDev]

I think that having a sitemap.xml is a best practice. A super fresh sitemap can help get your pages get indexed sooner. My sitemap.xml page keeps me focused on getting all of my content indexed. These days I have a custom script that updates my sitemap file daily. So there is no reason to not “keep up with” the sitemap. I don’t know why folks upload their sitemap into GWT. Seems like a better idea to have it in the root.

@Sgt_Kickaxe, what are you doing where your sitemap would impact your SQL database? Did you generating the sitemap.xml every time the page was loaded? I don’t get it unless your sitemap had a million or more entries.

IMO removing a sitemap will have zero impact on pages already in the index.

[tedster]

From my experience working with sites of many different types, I think that xml sitemaps are a best practice for those sites with many hundreds of thousands of real URLs, especially if there is a lot of URL churn. But for even a couple of thousand pages, the xml Sitemap has never been my choice.

If a site of that size cannot get decent indexation with their inherent information architecture and link structure, then they need to know that - and upgrade those structures. A Sitemap can cover over the problem without really fixing it.

So for the first type of major size, high churn site I'd say the risks of removing a Sitemap are pretty high. For a couple of thousand pages, removing a Sitemap is probably not going to be a big deal.

[atlrus]

Well, I am not the biggest fan of sitemaps, but I do use it occasionally, often when I try to test something related to what was once known as the "sand box" :)

They are great to get your website indexed quickly and to see how it gets indexed, but I have found that after the initial "burst" of crawling activity, Google gets into a routine and crawls your website in regular intervals, regardless of when you re-submitted the sitemap.

Of course, there are exceptions, such as news sitemaps or your website gets updated daily.

[breaker123]

If you manage a video site or a site that emphasizes videos then the XML sites maps can be invaluable because you can specify the thumb nail and then have your video thumb nail displayed in the SERPS.

Our XML site maps recently stopped adding new content and none of those videos display thumbs in SERPS. While we might rank #1 the lack of thumb nail will of course greatly reduce CTR.

[bloupbloup]

I see that most of the people here know their subject.

In my experience:
1. First of all, I have witnessed that xml sitemaps are indexed in google and appear in search results. You can witness it by typing: "site:*/sitemap.xml" in google. Today, we have not one sitemap but several several sitemaps (post,page,image,news). So, it is more garbage for our results. Knowing they don't have added value from content point of view, i wonder what is the impact for our indexing. Let's remind you that when googlebot indexes files, it values content. However there is no content in xml sitemaps but only links. I wonder if google considers it as a regular xml file in the context of pagerank. If it is the case, it is really bad, because xml sitemaps are going to have a negative impact since there is no real content in it.

2. I installed the module called phpbb seo to my phpbb forum. At the beginning, i installed this module that add a URL with the topic name in it rather than "viewforum.php"
So i had two URL for each page. "viewforum.php" version of the URL and "/my-topic-forum-name/". I also had a sitemap listing the URLS with topic names. I wanted to have the topic names URLs to be index and NOT the viewforum.php?f=#*$!
However google indexed both. It did not take account of the URLs in the XML sitemap.
Btw, i solved this issue with another module that did 301 redirect from viewforum to the topic name. So xml sitemap does not indicate which URLs to index.

[bloupbloup]

3. Before i had sitemaps. And i used to look into the number of submitted pages versus the number of index pages in webmaster tools. I did not witness any increase in indexing after the submission of the sitemap. Everyone can check that. Submit xml sitemap, check it once again after 3 days, you will see there will be no change. However in the log, googlebot visits those files. However, i don't see the benefit of it.

I double checked in google results. XML sitemaps did not help to index any URLs faster.

4. for people who have a news website with lots of posts, they don't need xml sitemaps. They just need to link their twitter account and post a twitter automatically when they submitted a new article. they may also submit this article to other social medias. They may use a solution like networkedblogs. That would be a way better impact than using a sitemap.

5. by the past, i also thought that XML sitemaps helped to get rid of parameters at the end of URL and provide the right URL to googlebot. I have understood it also did nothing. I thought sitemap would prevent to visit URL i did not wantand do the canonisation. I was also wrong. In fact, i discovered that there are so many urban legends around XML sitemaps. that's why i extensively tested it but i have never been able to find any benefits.


I guess there is some politics involved and the sitemap team at google wants to keep their job. I guess it would be more beneficial if they would work on valuing pings to social medias and get rid of xml sitemaps.

[piatkow]

Purely subjective: Site I had 5 years ago seemed to benefit when I created a site map. Current site never had one and is doing just as well.
Of course with Google the goal posts run on very well greased wheels.

[rango]

My experience is that having a sitemap definitely helps urls get indexed, particularly if that url was previously blocked from Google.

The other major benefit of an accurate sitemap is the stats provided by WMT on what percentage of the urls has been indexed. And any crawl issues are highlighted quickly.

[tedster]

What I've noticed is that for sites that have a high churn rate and publish or change a lot of content regularly, a regularly updated sitemap can speed indexing and that can be a good thing. And for sites with some kind of crawling issue, the same can happen - but in that case, I'd rather know that there is a crawling issue than patch it over with a sitemap.

[anallawalla]

Like Tedster said, A sitemap.xml begins to help a site with a large page count -- in the millions. I once managed a site with 74 million URLs. It was never fully crawled and indexed, but the priority hint values helped to ensure that the higher level (e.g. category, city etc) pages were picked up first.