|Is Sitemap.xml a must?|
Is Sitemap.xml a must for all websites? I didnt find a /sitemap.xml for sites like youtube, metacafe, digg, delicious. For google, msn I did find one though.
Although sitemaps are not a must, it is always nice having a sitemaps page for your users, so they can find important areas and pages of your site in just one page.
As for XML Sitemaps that can be submitted to Google, Yahoo! and MSN, I recommend you create one or more using Google's approach. Log onto Google Webmaster Tools and find out how you can easily create sitemaps using the protocol they recommended. Sitemaps are nice, especially for Google, because they sometimes check it once every 1-2 days and scan for pages that are not yet indexed in the Google Search index directory. This will help Google index your pages faster, as I noticed with my own site.
Within a week, I had 98% of my URLs in that XML sitemap file indexed my Google.
Now you don't have to create XML Sitemaps if you don't want to, but they can certainly speed things up a bit. It's up to you. If I were you, I would not code the XML file by hand. What I mean is, use PHP and parse that file as an XML file. Use PHP and while and for loops and scan your database for all your records and with that you can loop over all your content and dynamically create the necessary sitemap XML tags. This way you don't have to waste your time listing all your pages by hand. Let PHP or any other Server Side Language you might be using do the dirty work.
The site for which I am concerned is already having over 1,00,000 pages indexed by the search engines. And as I know they already maintain a nice index of its own - every time they crawl they update the same.
Maybe if we design a sitemap.xml file wont that get affected? Or the pages will get updated as per specify (say on an hourly basis for pages which are constantly updated) which can be benificial as well.
|The site for which I am concerned is already having over 1,00,000 pages indexed by the search engines. And as I know they already maintain a nice index of its own - every time they crawl they update the same. |
In your case, I don't think your site would need a sitemap just to make sure all pages are already indexed by Google. As you said, it seems like Google has already done the whole job for you, as it usually does with every major site out there. In my case however, I started using XML Sitemaps with Google ever since I put my website first online, so I can't really compare how different it is from using a sitemap or just letting Google do that for you, with no sitemap.
|Or the pages will get updated as per specify (say on an hourly basis for pages which are constantly updated) which can be benificial as well. |
Exactly, that is precisely how beneficial Sitemaps could be. According to their Sitemap Protocol specification, each URL has a tag specifying how often it is updated. Whether it is daily, weekly, monthly or yearly. It's a nice feature and I think it could certainly help update your indexed pages more often. I can't guarantee it will increase your Google Page Rank, but it is certainly a possibility.