Welcome to WebmasterWorld Guest from 23.20.241.155

Message Too Old, No Replies

Google Releases New Sitemap Generator Tool

     
11:05 am on Jan 14, 2009 (gmt 0)

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member



While most Sitemap generators either crawl websites or list the files on a server, we have created a different kind of Sitemap generator that uses several ways to find URLs on your website and then allows you to automatically create and maintain different kinds of Sitemap files.

A new Google Sitemap Generator for your website
2009-01-13 - [googlewebmastercentral.blogspot.com...]

5:38 pm on Jan 14, 2009 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



It's a server plug-in, so it requires admin level server access - but it does work for both Apache and IIS. Lots of user options here as well as three different methods for discovering urls on the site. It sounds like a well-thought out project.
7:03 pm on Jan 14, 2009 (gmt 0)

10+ Year Member



Good to know that the $25 i spent on < a commercial sitemap generator > (for like the tenth time) the other day was wasted.

(It's not really about the money - $25 - rather I'm mad at myself for constantly paying for things I could get for free. Oh well).

[edited by: Robert_Charlton at 7:54 pm (utc) on Jan. 14, 2009]
[edit reason] edited specific [/edit]

7:16 pm on Jan 14, 2009 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



if your 404 page is returning a 200 status code your competition could use this to stuff your google sitemap with errors. i am not even going to get into the fun you could have with sites that generate urls on the fly. oh the potential chaos that a disobedient bot could cause on certain websites in selective situations.
7:23 pm on Jan 14, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I thought the point of a sitemap was to notify search engines of content that is not being found. Ie - content that is not showing up in our log files.

Though I can see where it would help expose content that other search engines are sending traffic to... It doesn't beat building your own sitemap tool... at least in my opinion.

8:17 pm on Jan 14, 2009 (gmt 0)

5+ Year Member



Goodroi,

Regarding your comment:
"...the fun you could have with sites that generate urls on the fly. oh the potential chaos that a disobedient bot could cause on certain websites in selective situations. "

Can you please explain further? Most of my sites are generating urls on the fly and indeed I am getting google sitemaps errors with urls that don't exist and just go to a custom 404 page. The url rewrite was done for SEO purposes...but definitely I don't want this to create another problem and most importantly an opportunity for malicious competitors and bots to affect our ranking.

How can a bot or a malicious competitor create this havoc and how can I stop this?

Thanks,

KH

9:40 pm on Jan 14, 2009 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



What they should do, incorporate google analytics, google-bot and any other bots with google webmasters. Pull the url's from there dump them into a space in google webmaster tools and give the option to the webmaster to review them to create their own site map.
2:50 am on Jan 15, 2009 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



hi KHWeb,

this new sitemap generator tool creates a sitemap file based on the urls that receive traffic. if someone wants to, they can send you fake traffic to any url. you may not want those urls automatically included in your sitemap. this new sitemap generator will include those urls since it is based on traffic. i dont want to go much more into it because i dont want to encourage people to go looking for trouble.

a clever person in just the right situation could have fun. imho the potential for trouble here is limited so you should not be very worried.

7:04 pm on Jan 15, 2009 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



if someone wants to, they can send you fake traffic to any url. you may not want those urls automatically included in your sitemap.

why can't you robots.txt those urls?

7:12 pm on Jan 15, 2009 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



why can't you robots.txt those urls?

whoops! just answered my own question; i haven't installed it, but the doc reads the following

Under URL query fields, read the special privacy notice and then specify URL query fields that you want to include in Sitemaps. Because Google Sitemap Generator removes all query fields from URLs in order to protect user privacy, you must explicitly specify any query field that you want URLs to include.
5:56 pm on Jan 20, 2009 (gmt 0)

10+ Year Member



i installed it on our corporate IIS 6 running Winblows Server 2003
HttpRewrite & IIS Zip Compression enabled on the box

After installation wouldn't run properly due to mis-configured permissions. I set the correct permissions, after which it gave me another error and i just gave up and uninstalled...

The search for a perfect sitemap generator continues...

6:13 pm on Jan 20, 2009 (gmt 0)

WebmasterWorld Senior Member quadrille is a WebmasterWorld Top Contributor of All Time 10+ Year Member



What about us technophobes?

Why can't Google cook up a reliable online sitemap generator, that does not have artificial linits (eg 500 pages), or fees, or a pageful of promos, or code that hides links to the 3rd party system?

Why does Google claim to want everyone to have a fair chance - then discriminate against the non-technical?

I don't expect answers, BTW; I've said similar on Google's group, and was told "Why would they offer a tool that spiders, when they'll be spidering the site anyway" - so why bother with sitemaps at all?

7:48 pm on Jan 28, 2009 (gmt 0)

5+ Year Member



If you run your own server (or at least have root) this is not that difficult to install - I ran into a couple of snags but the Google Group for this tool was enough to get me going.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month