- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Bots using robots
cooldoug - 1:29 am on Mar 9, 2005 (gmt 0)
Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?
Thread source: http://www.webmasterworld.com/robots_txt/567.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com