dumbfounder - 5:48 pm on Jul 15, 2004 (gmt 0)

The Contractor,

Either is fine. I do a case-insensitive search for "dumbot" anywhere on any line that starts with "user-agent" (again, case insensitive). Then any disallows found after that, up until the next user-agent line, are attributed. Other than the pattern match, it is the same exact code I use for the user-agent "*". You are the only person to ever complain that I have not adhered to robots.txt and I have been doing this for 8 months now, which leads me to think that the problem is not a large one. The absolute best way to track down the problem is to use real examples from real websites, I cannot account for the infinite possibilities found out on the web. I think you are doing a disservice to others by refusing to give out your web address. You can email it to me if you wish. info@dumbfind.com. Or you can call me 703-683-3077, or you can drive to my house and hand me a letter (just do a whois on dumbfind.com).

thanks jd, how about if I put a link to google's bot information page at the end of mine? I would feel better about that than simply plagiarizing their content.