Welcome to WebmasterWorld Guest from 54.226.133.245

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt

robots.txt

     
1:21 pm on Feb 21, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 4, 2005
posts:180
votes: 0


If you dont have a robot.txt the search engines will just index your site, Why have the robots.txt file
1:25 pm on Feb 21, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 27, 2003
posts:159
votes: 0


You may not want certain/all search engines to see certain/all pages on your website.

Jon

8:43 pm on Feb 25, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 13, 2003
posts:1281
votes: 0


Robots.txt will help keep SEs out of directories you don't want indexed. For example, someone who sells his images might not want google's imagebot to pull all of his images out of a directory and list them where they can be freely downloaded.
8:59 pm on Feb 25, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


And having any robots.txt file --even a blank one-- will prevent a lot of 404-Not Found errors in your logs, thus making it easier to identify real problems.

Jim