homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

to map or not to map

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

Msg#: 4568754 posted 10:30 pm on Apr 28, 2013 (gmt 0)

One of those fuzzy philosophical questions:

Here in this bright and beautiful year of our lord 2013, when-if-ever is a sitemap necessary? I'm asking here about xml or equivalent sitemaps aimed at search engines, not at visible sitemaps for the benefit of humans.

The book says sitemaps are to help the search engine find pages they might otherwise miss. When does this happen?

Obviously any page linked by <a href... is out. Robots can find those just fine. As I understand it, a "nofollow" tag makes no difference; that just applies to indexing, not to physically locating a file. Links created by javascript used to be an issue, but the major search engines will happily execute scripts so that's another non-starter. Directories requiring logins might be a problem, but there's always the "satisfy any" directive-- and why are you indexing members-only pages anyway?

For comparison purposes: I used to have a pretty comprehensive sitemap. About a year back I said the heck with it and cut it back to only directory-index pages-- either files whose actual name* is index.html, or named pages performing the same function. So even if the server explodes and all sitewide navigation footers vanish into the ether, the search engine can still find everything.

* That is, ahem, the real name of the physical file. Not the URL.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved