incrediBILL - 6:03 pm on Oct 26, 2006 (gmt 0)

All of these tags per page are starting to bloat the heck out of the web and I'm getting quite sick of it.

How about sticking this stuff in robots.txt files, with ALTERNATE web tags, so our pages don't end up 3 miles long full of attributes and 100 bytes of actual content?

How hard would it be to do something civilized like:

User-agent: Slurp
Directive: NOODP
Directive: NOYDIR
Directive: NOARCHIVE
Disallow: /cgi-bin/
Disallow: /webstat/

Sure would be nice opposed to mucking up every single webpage.

Put more control over spiders back in robots.txt where it belongs and stop forcing us to clutter our pages site wide with the same silly tags unless we WANT that level of granular control.