Forum Moderators: open
What's it up to?
I always thought it was looking for redirects from 404 errors.
By looking at the response for random pages it is possible to work out how your web is set up, have also seen GBot doing this