Google Releases New Sitemap Generator Tool

[pageoneresults]

While most Sitemap generators either crawl websites or list the files on a server, we have created a different kind of Sitemap generator that uses several ways to find URLs on your website and then allows you to automatically create and maintain different kinds of Sitemap files.

A new Google Sitemap Generator for your website
2009-01-13 - [googlewebmastercentral.blogspot.com...]

[tedster]

It's a server plug-in, so it requires admin level server access - but it does work for both Apache and IIS. Lots of user options here as well as three different methods for discovering urls on the site. It sounds like a well-thought out project.

[notsosmart]

Good to know that the $25 i spent on < a commercial sitemap generator > (for like the tenth time) the other day was wasted.

(It's not really about the money - $25 - rather I'm mad at myself for constantly paying for things I could get for free. Oh well).

[edited by: Robert_Charlton at 7:54 pm (utc) on Jan. 14, 2009]
[edit reason] edited specific [/edit]

[goodroi]

if your 404 page is returning a 200 status code your competition could use this to stuff your google sitemap with errors. i am not even going to get into the fun you could have with sites that generate urls on the fly. oh the potential chaos that a disobedient bot could cause on certain websites in selective situations.

[maximillianos]

I thought the point of a sitemap was to notify search engines of content that is not being found. Ie - content that is not showing up in our log files.

Though I can see where it would help expose content that other search engines are sending traffic to... It doesn't beat building your own sitemap tool... at least in my opinion.

[KHWeb]

Goodroi,

Regarding your comment:
"...the fun you could have with sites that generate urls on the fly. oh the potential chaos that a disobedient bot could cause on certain websites in selective situations. "

Can you please explain further? Most of my sites are generating urls on the fly and indeed I am getting google sitemaps errors with urls that don't exist and just go to a custom 404 page. The url rewrite was done for SEO purposes...but definitely I don't want this to create another problem and most importantly an opportunity for malicious competitors and bots to affect our ranking.

How can a bot or a malicious competitor create this havoc and how can I stop this?

Thanks,

KH

[trinorthlighting]

What they should do, incorporate google analytics, google-bot and any other bots with google webmasters. Pull the url's from there dump them into a space in google webmaster tools and give the option to the webmaster to review them to create their own site map.

[goodroi]

hi KHWeb,

this new sitemap generator tool creates a sitemap file based on the urls that receive traffic. if someone wants to, they can send you fake traffic to any url. you may not want those urls automatically included in your sitemap. this new sitemap generator will include those urls since it is based on traffic. i dont want to go much more into it because i dont want to encourage people to go looking for trouble.

a clever person in just the right situation could have fun. imho the potential for trouble here is limited so you should not be very worried.

[nealrodriguez]

if someone wants to, they can send you fake traffic to any url. you may not want those urls automatically included in your sitemap.

why can't you robots.txt those urls?

[nealrodriguez]

why can't you robots.txt those urls?

whoops! just answered my own question; i haven't installed it, but the doc reads the following

Under URL query fields, read the special privacy notice and then specify URL query fields that you want to include in Sitemaps. Because Google Sitemap Generator removes all query fields from URLs in order to protect user privacy, you must explicitly specify any query field that you want URLs to include.

[papachumba]

i installed it on our corporate IIS 6 running Winblows Server 2003
HttpRewrite & IIS Zip Compression enabled on the box

After installation wouldn't run properly due to mis-configured permissions. I set the correct permissions, after which it gave me another error and i just gave up and uninstalled...

The search for a perfect sitemap generator continues...

[Quadrille]

What about us technophobes?

Why can't Google cook up a reliable online sitemap generator, that does not have artificial linits (eg 500 pages), or fees, or a pageful of promos, or code that hides links to the 3rd party system?

Why does Google claim to want everyone to have a fair chance - then discriminate against the non-technical?

I don't expect answers, BTW; I've said similar on Google's group, and was told "Why would they offer a tool that spiders, when they'll be spidering the site anyway" - so why bother with sitemaps at all?

[civgroup]

If you run your own server (or at least have root) this is not that difficult to install - I ran into a couple of snags but the Google Group for this tool was enough to get me going.