homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Queries about Robots.txt
Importance of Robots.txt

5+ Year Member

Msg#: 3787861 posted 12:39 pm on Nov 17, 2008 (gmt 0)

Hello viewers,

A warm greetings to you all.....

As I am new to this forum, I have least knowledge about search engine.....So i would like to know what is Robots.txt?

What is its function? How does it work? Please tell be about the importance of Robots.txt in terms of search engine? How far is it helpful?

Looking ahead for your valuable comments





WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3787861 posted 5:24 pm on Nov 17, 2008 (gmt 0)

hello monalisa,

robots.txt is a text file that is placed at the root level of a website. its purpose is to help inform automated programs aka search engine spiders which areas of your website they should access or not access. search engines normally assume they can spider anything online so unless you specifically block it they will probably try to access it. please remember that robots.txt is a voluntary protocol. if an automated program is misbehaving (because of a mistake or intentional programming) then it will still access areas of your site that are blocked by robots.txt.

some webmasters of simple websites that only contain files they want included in search engines do not even upload a robots.txt. most webmasters use robots.txt to block search engines from accessing folders containing stats, user data, scripts and other things you do not want listed in search engines.

if you want to properly secure and protect those files you should use an htaccess file or isapi. htaccess allows webmasters to block access compared to robots.txt which simply asks spiders to be nice and not index certain areas.

you may also want to read this old thread on the same topic

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved