- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Robozilla and Robots.txt
Brett_Tabke - 12:23 pm on May 3, 2001 (gmt 0)
Does robozilla obey robots.txt? And if so, what does it do if it runs into a url in the database, that is also blocked by a robots.txt at the site?
Thread source: http://www.webmasterworld.com/robots_txt/111.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com