Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt standard


Son_House - 3:17 am on Dec 7, 2001 (gmt 0)


I would use your first example. I believe by using the User-agent names of scooter and slurp those bots will follow what is specifically written for them and disregard anything written for User-agent: *


Thread source:: http://www.webmasterworld.com/robots_txt/124.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com