Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robozilla and Robots.txt
Brett_Tabke - 12:23 pm on May 3, 2001 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/111.htm
Does robozilla obey robots.txt? And if so, what does it do if it runs into a url in the database, that is also blocked by a robots.txt at the site?
Brought to you by WebmasterWorld: http://www.webmasterworld.com