Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Google Link points and Robots.txt


Xoc - 8:06 pm on Mar 4, 2001 (gmt 0)


Brett-

I think you misread how the a robot is supposed to parse the LookSmart robots.txt (or I misread what you said). The Disallow doesn't have anything after it. So that means that that particular spider is allowed to spider the web site. All non-identified spiders that weren't specifically listed get hit by the Disallow: *

So Googlebot should be allowed to spider LookSmart. The Snap robots.txt, though, disallows everything.


Thread source:: http://www.webmasterworld.com/robots_txt/164.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com