Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Yahoo! Slurp Now Supports Wildcards in robots.txt


bouncybunny - 1:29 am on Nov 7, 2006 (gmt 0)


I'm trying to work out if and how this differs from Google's usage? And MSNs for that matter.

If all we are interested is these three bots (and for many that may be the case) then using;

User-agent: *

should be enough now? Yes? No?


Thread source:: http://www.webmasterworld.com/robots_txt/3144662.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com