Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Queries about Robots.txt

Importance of Robots.txt

12:39 pm on Nov 17, 2008 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 26, 2007
votes: 0

Hello viewers,

A warm greetings to you all.....

As I am new to this forum, I have least knowledge about search engine.....So i would like to know what is Robots.txt?

What is its function? How does it work? Please tell be about the importance of Robots.txt in terms of search engine? How far is it helpful?

Looking ahead for your valuable comments



5:24 pm on Nov 17, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
votes: 280

hello monalisa,

robots.txt is a text file that is placed at the root level of a website. its purpose is to help inform automated programs aka search engine spiders which areas of your website they should access or not access. search engines normally assume they can spider anything online so unless you specifically block it they will probably try to access it. please remember that robots.txt is a voluntary protocol. if an automated program is misbehaving (because of a mistake or intentional programming) then it will still access areas of your site that are blocked by robots.txt.

some webmasters of simple websites that only contain files they want included in search engines do not even upload a robots.txt. most webmasters use robots.txt to block search engines from accessing folders containing stats, user data, scripts and other things you do not want listed in search engines.

if you want to properly secure and protect those files you should use an htaccess file or isapi. htaccess allows webmasters to block access compared to robots.txt which simply asks spiders to be nice and not index certain areas.

you may also want to read this old thread on the same topic


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members