Sitemap.xml : Hide from Public view?

[donna130]

Hi. I posted a similar topic recently. However, it applies much more to sitemap.xml file than any other file.

I'm looking for a way/ways for me/other websmasters to SHOW (obviously) the sitemap to Google's (and otehr SE) webbots, but to HIDE it from other public (competitors!) viewing. Otherwise it seems like we're essentially paying for our competitors' spyfu etc accounts.

I don't want our competitors to see our whole list of keywords, which are built into the URLs. It's a very common thing for webmasters to do this. It's like handing over your entire private list of kw and competitive business information to your competitors. Otherwise, it's like having all your private bank accounts and pins :) posted publicly.

Thanks for your solutions.

[phranque]

it's easy to spider a site to get a list of all the urls and titles.
you might consider responding with a 403 based on user agent but user agent spoofing is also fairly simple.
if you want to get more sophisticated, the Search Engine Spider and User Agent Identification forum Charter [webmasterworld.com] has information about verifying Googlebot and it is a similar process for the others.

[donna130]

Cool. I'll check it out, thanks. I'm always interested in new suggestions and ideas and hope others too will contribute their work-arounds to this common problem Thanks again.

[downhiller80]

For what it's worth I give my sitemap a name that no-one's going to be able to guess, rather than sitemap.xml, then just submit it to the three big SEs.

[donna130]

AWESOME! Yeah, I thought I saw something like that a few weeks ago...but I guess it sort of went "in one eye and out the other". Hey, thanks. Sometimes it's those simple ideas that are the best. Obviously, we gotta understand (I know you do, but for other readers I mean), that, as phranque said, they can still spider your site and then manually (more likely with a macro) search each of those URLs one-by-one, but right now, yours is the best idea. As always I'm open to new ideas, and it would be cool to see additional solutions here. Thanks so much.

[downhiller80]

It's not massively clever, but it is very simple, and certainly better than just leaving it named sitemap.xml. I just have a random "keyboard mash" sring as part of the filename :)