homepage Welcome to WebmasterWorld Guest from 54.163.91.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
What is a Robots.txt?
Noximus




msg:1525931
 3:35 pm on Oct 28, 2002 (gmt 0)

Sorry for an easy question but i don't have big experience being a webmaster, and i would like to know what is robots.txt?

thanks for the answer in advance
Noximus

[edited by: WebGuerrilla at 3:53 pm (utc) on Oct. 28, 2002]
[edit reason] Split from another thread [/edit]

 

soapystar




msg:1525932
 3:46 pm on Oct 28, 2002 (gmt 0)

just search for robots.txt on google.......erm...now why didnt I think of that?!

SlyGuy




msg:1525933
 3:47 pm on Oct 28, 2002 (gmt 0)

Noximus :

Robots.txt [searchengineworld.com]

- Chad

QNetwork




msg:1525934
 3:48 pm on Oct 28, 2002 (gmt 0)

robots.txt is a config file generally located in the root directory of web site. There are certain parameters which specify how a search engine robot (or any other robot) will crawl the site. You can specify some directories or file not to be crawled by. You can also allow/disallow a specific robot (identified by user agent). Search by "robots.txt" in this site, you will find lots of important information.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved