Forum Moderators: Robert Charlton & goodroi
When a request comes to your server and says that it is from googlebot, your server takes this 2-step approach in deciding how to respond:
1. REVERSE LOOK-UP
Using the IP address that request, your server does a reverse DNS lookup. Because Googlebots have reverse DNS enabled, the result should be a subdomain of googlebot.com. If it isn't, then your server does not return the content -- it's someone spoofing the googlebot user agent and your server can return a 403 Forbidden status.2. FORWARD LOOK-UP
To make sure that the reverse-DNS entry isn't fake, your server now uses the full googlebot subdomain that was reported in step 1 and does a forward DNS look up. That should return the same IP address that the original request. Again, if it isn't, your server doesn't return the content.
In the second link I gave, incrediBILL offers some other practical tips for employing this method - be sure to check them out.