Welcome to WebmasterWorld Guest from


Charter - Sitemaps, Meta Data, and robots.txt

Forum: Sitemaps, Meta Data, and robots.txt
Category: The Search Engine World
Moderator: goodroi
Previous Moderator: ThomasB

The robots.txt exclusion standard is a method that some search engines will follow when building automated robots to mine the web for data.

Posting Topics:
Topics directly related to robots.txt

Related Robots.txt Info from Search Engine World

  • Robots.txt Tutorial [webmasterworld.com] with examples.
  • Robots Exclusion Meta Tag [webmasterworld.com] Using robots metatags.
  • Robots Exclusion Standard [webmasterworld.com] The current working draft proposal for the robots exclusion standard.
  • Robots.txt examples. You can use and modify (rename to robots.txt before using):

    1. robots1.text [webmasterworld.com] allows all robots.
    2. robots2.text [webmasterworld.com] bans/disallows all robots.
    3. robots3.text [webmasterworld.com] bans/disallows robots from your cgi-bin and images directories.
    4. robots4.text [webmasterworld.com] a comprehensive robots.txt file that only allows known "nice guy" spiders.

External Robots.txt Resource