Welcome to WebmasterWorld Guest from 54.221.28.179

Forum Moderators: goodroi

Message Too Old, No Replies

Queries about Robots.txt

Importance of Robots.txt

     

Monalisa

12:39 pm on Nov 17, 2008 (gmt 0)

5+ Year Member



Hello viewers,

A warm greetings to you all.....

As I am new to this forum, I have least knowledge about search engine.....So i would like to know what is Robots.txt?

What is its function? How does it work? Please tell be about the importance of Robots.txt in terms of search engine? How far is it helpful?

Looking ahead for your valuable comments

Thanks

Monalisa

goodroi

5:24 pm on Nov 17, 2008 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



hello monalisa,

robots.txt is a text file that is placed at the root level of a website. its purpose is to help inform automated programs aka search engine spiders which areas of your website they should access or not access. search engines normally assume they can spider anything online so unless you specifically block it they will probably try to access it. please remember that robots.txt is a voluntary protocol. if an automated program is misbehaving (because of a mistake or intentional programming) then it will still access areas of your site that are blocked by robots.txt.

some webmasters of simple websites that only contain files they want included in search engines do not even upload a robots.txt. most webmasters use robots.txt to block search engines from accessing folders containing stats, user data, scripts and other things you do not want listed in search engines.

if you want to properly secure and protect those files you should use an htaccess file or isapi. htaccess allows webmasters to block access compared to robots.txt which simply asks spiders to be nice and not index certain areas.

you may also want to read this old thread on the same topic
[webmasterworld.com...]

 

Featured Threads

Hot Threads This Week

Hot Threads This Month