Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- wildcards in page names

Sanenet - 2:01 pm on Jun 16, 2005 (gmt 0)


* doesn't work as a wildcard in robots.txt

the syntax to disallow a directory is simply:

UserAgent:*
Disallow:/store/scripts/

Which would block everybody from indexing anything in scripts.

Disallow: /store/scripts/emailFriend.asp*
Disallow: /store/scripts/contactUs.asp?emailSubject*
Is either going to be ignored, or will lead to just those two pages being ignored.

www.robotstxt.org

Thread source:: http://www.webmasterworld.com/robots_txt/661.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com