Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt question


wilderness - 7:33 pm on Mar 2, 2003 (gmt 0)


Javi,
Welcome to Webmaster World.

It is imperative that you keep in mind that robots.txt is a SUGGESTION to bots of compliance.
Many bad bots don't even read robots.txt.

With the above in mind. . .

User-agent: *
Disallow: /

denies all bots.
I seem to recall that Allow is not valid protocol in robots'txt.

So your alyernatives are either to deny all or list each "1 by 1"


Thread source:: http://www.webmasterworld.com/robots_txt/138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com