Welcome to WebmasterWorld Guest from 54.162.12.134

Forum Moderators: goodroi

Message Too Old, No Replies

What is a Robots.txt?

     
3:35 pm on Oct 28, 2002 (gmt 0)

10+ Year Member



Sorry for an easy question but i don't have big experience being a webmaster, and i would like to know what is robots.txt?

thanks for the answer in advance
Noximus

[edited by: WebGuerrilla at 3:53 pm (utc) on Oct. 28, 2002]
[edit reason] Split from another thread [/edit]

3:46 pm on Oct 28, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



just search for robots.txt on google.......erm...now why didnt I think of that?!
3:47 pm on Oct 28, 2002 (gmt 0)

10+ Year Member



Noximus :

Robots.txt [searchengineworld.com]

- Chad

3:48 pm on Oct 28, 2002 (gmt 0)

10+ Year Member



robots.txt is a config file generally located in the root directory of web site. There are certain parameters which specify how a search engine robot (or any other robot) will crawl the site. You can specify some directories or file not to be crawled by. You can also allow/disallow a specific robot (identified by user agent). Search by "robots.txt" in this site, you will find lots of important information.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month