Page is a not externally linkable
tangor - 7:18 am on Aug 16, 2011 (gmt 0)
Correct, the regex $ is not required. Done. Otherwise, it correct. Best method is to disallow ALL BOTS then list which bots ARE ALLOWED, but that put me in the minority (called whitelisting)...