Msg#: 2613 posted 11:48 am on Dec 13, 2005 (gmt 0)
What is robot.txt. I am confused i have been doing the site for a year now and have a feeling the reason my site is badly performing in the SE's is the robopt.txt has not been done. Also what is the best way of making you sitemap as i have done an xml sitemap with a third party tool and dont think it is doing the right job.? Any help much appreciated Very keen, Youngkeen
Msg#: 2613 posted 12:45 pm on Dec 13, 2005 (gmt 0)
robots.txt is simply a text file that you place in the root of your web folder that gives instructions to spiders such as Googles Googlebot - [google.com...] or Yahoos Slurp - [help.yahoo.com...] what it can and can't crawl.
By default if you don't have a robots.txt it assumes everything on your site is freely crawable, if it is you don't have the need for a robots file. If however you don't want it crawling a certain page you can indicate this in the file.
This page has more details (hope it is ok to list, sure i have seen it here before) [robotstxt.org...]
But to answer your quesion, the lack of a robots.txt file won't be hurting your rankings.