Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Bots ignoring my robots.txt


jdMorgan - 3:02 am on Nov 5, 2003 (gmt 0)


roddy,

May I suggest:

User-agent: *
Disallow: /privmsg.php
Disallow: /search.php
Disallow: /faq.php
Disallow: /memberlist.php
Disallow: /groupcp.php
Disallow: /profile.php
Disallow: /login.php
Disallow: /posting.php
Disallow: /viewonline.php

Ref: [robotstxt.org...]

The robots.txt validator -- like most other validators -- indicates that the 'code' is valid, and not that it will do what you desire it to do.

Disallowing /xyz.php will also disallow /xyz.php?anything

Jim


Thread source:: http://www.webmasterworld.com/robots_txt/184.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com