homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt
robots.txt
indodpendo




msg:1528954
 1:21 pm on Feb 21, 2005 (gmt 0)

If you dont have a robot.txt the search engines will just index your site, Why have the robots.txt file

 

phidentity




msg:1528955
 1:25 pm on Feb 21, 2005 (gmt 0)

You may not want certain/all search engines to see certain/all pages on your website.

Jon

SEOMike




msg:1528956
 8:43 pm on Feb 25, 2005 (gmt 0)

Robots.txt will help keep SEs out of directories you don't want indexed. For example, someone who sells his images might not want google's imagebot to pull all of his images out of a directory and list them where they can be freely downloaded.

jdMorgan




msg:1528957
 8:59 pm on Feb 25, 2005 (gmt 0)

And having any robots.txt file --even a blank one-- will prevent a lot of 404-Not Found errors in your logs, thus making it easier to identify real problems.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved