Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
BGumble - 4:03 pm on May 7, 2003 (gmt 0)
Do you exclude crawlers that produce few results like WiseNut or AltaVista? Or let them through hoping they might one day become relevant. On a large site, extraneous crawlers can chew lots of bandwidth.