goodroi - 1:32 pm on Jul 3, 2010 (gmt 0)

wildcards aka pattern matching is not officially part of the robots.txt protocol. this means most of the big search engines support it but most of the smaller one won't.


According to Google's page [google.com...]

To match a sequence of characters, use an asterisk (*). For instance, to block access to all subdirectories that begin with private: User-agent: Googlebot Disallow: /private*/