Welcome to WebmasterWorld Guest from 54.159.50.111

Forum Moderators: goodroi

Message Too Old, No Replies

What is a Robots.txt?

     
3:35 pm on Oct 28, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 7, 2002
posts:403
votes: 0


Sorry for an easy question but i don't have big experience being a webmaster, and i would like to know what is robots.txt?

thanks for the answer in advance
Noximus

[edited by: WebGuerrilla at 3:53 pm (utc) on Oct. 28, 2002]
[edit reason] Split from another thread [/edit]

3:46 pm on Oct 28, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 19, 2002
posts:1945
votes: 0


just search for robots.txt on google.......erm...now why didnt I think of that?!
3:47 pm on Oct 28, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 18, 2001
posts:397
votes: 0


Noximus :

Robots.txt [searchengineworld.com]

- Chad

3:48 pm on Oct 28, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 20, 2002
posts:81
votes: 0


robots.txt is a config file generally located in the root directory of web site. There are certain parameters which specify how a search engine robot (or any other robot) will crawl the site. You can specify some directories or file not to be crawled by. You can also allow/disallow a specific robot (identified by user agent). Search by "robots.txt" in this site, you will find lots of important information.