Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe and Support WebmasterWorld
robots.txt robots.txt indodpendo msg:1528954 1:21 pm on Feb 21, 2005 (gmt 0) If you dont have a robot.txt the search engines will just index your site, Why have the robots.txt file
phidentity msg:1528955 1:25 pm on Feb 21, 2005 (gmt 0)
You may not want certain/all search engines to see certain/all pages on your website.
SEOMike msg:1528956 8:43 pm on Feb 25, 2005 (gmt 0)
Robots.txt will help keep SEs out of directories you don't want indexed. For example, someone who sells his images might not want google's imagebot to pull all of his images out of a directory and list them where they can be freely downloaded. jdMorgan msg:1528957 8:59 pm on Feb 25, 2005 (gmt 0)
And having any robots.txt file --even a blank one-- will prevent a lot of 404-Not Found errors in your logs, thus making it easier to identify real problems.