Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Bots ignoring my robots.txt


roddy - 12:17 pm on Nov 3, 2003 (gmt 0)


Looks useful, but this will prevent google crawling ALL dynamic pages - I only want to prevent crawling of certain pages.

(actually I'm quite happy to treat all bots the same, but Google is the only one taking any significant bandwidth)

Roddy


Thread source:: http://www.webmasterworld.com/robots_txt/184.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com