Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

What is a Robots.txt?



3:35 pm on Oct 28, 2002 (gmt 0)

10+ Year Member

Sorry for an easy question but i don't have big experience being a webmaster, and i would like to know what is robots.txt?

thanks for the answer in advance

[edited by: WebGuerrilla at 3:53 pm (utc) on Oct. 28, 2002]
[edit reason] Split from another thread [/edit]


3:46 pm on Oct 28, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

just search for robots.txt on google.......erm...now why didnt I think of that?!


3:47 pm on Oct 28, 2002 (gmt 0)

10+ Year Member

Noximus :

Robots.txt [searchengineworld.com]

- Chad


3:48 pm on Oct 28, 2002 (gmt 0)

10+ Year Member

robots.txt is a config file generally located in the root directory of web site. There are certain parameters which specify how a search engine robot (or any other robot) will crawl the site. You can specify some directories or file not to be crawled by. You can also allow/disallow a specific robot (identified by user agent). Search by "robots.txt" in this site, you will find lots of important information.

Featured Threads

Hot Threads This Week

Hot Threads This Month