Page is a not externally linkable
- Search Engines
-- Directories
---- New LookSmart Homepage


Quinn - 4:25 pm on Feb 7, 2003 (gmt 0)


They don't want anyone spidering their links. Robots.txt shows :
User-agent: *
Disallow:

Webwalla,

This is what I see in the robots.txt file..

Disallow: /category/be_zealous.jhtml
Disallow: /category/manage.jhtml
Disallow: /category/pending_websites.jhtml

This would allow bots into then category folder where the pages in the directory reside.

<edited by Quinn> Sorry Webwalla, just realized that you and skibum were referring to looksmart, not zeal</edit>


Thread source:: http://www.webmasterworld.com/directories/1350.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com