Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Why robots.txt if you want everything indexed?


Larry_C - 5:56 pm on Mar 2, 2003 (gmt 0)


Hi everyone,

I've lurked for awhile and tried to find the answer to this question. If you want everything in your site indexed - does it even matter if you have a robot.txt file? I realize that you'll get a 404 error for each time, but do the spiders care?

Thanks for helping out a webmaster-wannabe!

Larry


Thread source:: http://www.webmasterworld.com/robots_txt/33.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com