Welcome to WebmasterWorld Guest from 54.227.125.200

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt

robots.txt

     

indodpendo

1:21 pm on Feb 21, 2005 (gmt 0)

10+ Year Member



If you dont have a robot.txt the search engines will just index your site, Why have the robots.txt file

phidentity

1:25 pm on Feb 21, 2005 (gmt 0)

10+ Year Member



You may not want certain/all search engines to see certain/all pages on your website.

Jon

SEOMike

8:43 pm on Feb 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Robots.txt will help keep SEs out of directories you don't want indexed. For example, someone who sells his images might not want google's imagebot to pull all of his images out of a directory and list them where they can be freely downloaded.

jdMorgan

8:59 pm on Feb 25, 2005 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



And having any robots.txt file --even a blank one-- will prevent a lot of 404-Not Found errors in your logs, thus making it easier to identify real problems.

Jim

 

Featured Threads

Hot Threads This Week

Hot Threads This Month