Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- updating sitemaps - how often?
phranque - 10:38 am on Apr 17, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/4296144.htm
with 600,000 products, perhaps it would be better to create a script to generate the 12+ sitemaps from the product database instead of crawling the site.
Brought to you by WebmasterWorld: http://www.webmasterworld.com