ergophobe - 5:04 pm on Nov 29, 2007 (gmt 0)

The article is so vague on what exactly they're proposing [edit: thanks Jim.]. It would be nice if there was some regex capability (like the Google extensions, but a bit more).

Sitemaps vs robots.txt. As I see it

Sitemaps: "Crawl here. Index this. And if you want to follow other links, that's fine with me".

robots.txt: "Don't crawl here. Don't index this. And if you do or don't find stuff in other places, that's none of my business."

I don't think given that difference, one can ever "steal the lead" over the other really.