A little while back I saw someone hit my robots.txt with "DigExt" in their user agent. I'm always on the lookout for spiders as I can add a piece of their UA to a directive in my shopping cart software, and it will keep? and & out of the URL for them for easy spidering. I searched Google for DigExt and the first result was a WebmasterWorld thread which concluded that DigExt does mean robot. I added it to my software's directive, but I recently found out that a couple customers couldn't check out of my store and I tracked it back to their having "DigExt" in their UA. Just in case anyone thinks DigExt means robot... IT DOES NOT. :)
- Grant
