homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Charter : Sitemaps, Meta Data, and robots.txt


Charter - Sitemaps, Meta Data, and robots.txt

Forum: Sitemaps, Meta Data, and robots.txt
Category: The Search Engine World
Moderator: goodroi
Previous Moderator: ThomasB

The robots.txt exclusion standard is a method that some search engines will follow when building automated robots to mine the web for data.

Posting Topics:
Topics directly related to robots.txt

Related Robots.txt Info from Search Engine World

  • Robots.txt Tutorial [webmasterworld.com] with examples.
  • Robots Exclusion Meta Tag [webmasterworld.com] Using robots metatags.
  • Robots Exclusion Standard [webmasterworld.com] The current working draft proposal for the robots exclusion standard.
  • Robots.txt examples. You can use and modify (rename to robots.txt before using):

    1. robots1.text [webmasterworld.com] allows all robots.
    2. robots2.text [webmasterworld.com] bans/disallows all robots.
    3. robots3.text [webmasterworld.com] bans/disallows robots from your cgi-bin and images directories.
    4. robots4.text [webmasterworld.com] a comprehensive robots.txt file that only allows known "nice guy" spiders.

External Robots.txt Resource


Home / Forums Index / Search Engines / Charter : Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved