Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- After using robots.txt none of the bots are crawling my site! Help
penders - 9:33 am on Mar 3, 2010 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/4088929.htm
|I would go with a whitelist of allowed bots, disallowing all others... |
So you allow very few robots? What about all the other 'good bots'? - or do they not count anything towards site promotion?
Brought to you by WebmasterWorld: http://www.webmasterworld.com