Forum Moderators: open
what is the best way to identify googlebot?
are all googlebots using isp that looks like crawl-?-?-?-?.googlebot.com?
heard that googlebot can visit page in different ip or isp address to check if the content is the same you're displaying to ordinary user and googlebot.
Don't always trust the hostname (such as crawlerxx.googlebot.com) when trying to verify a Googlebot spider. I have seen this spoofed.
Check the whois records on the IP address. If it is registered to Google or Savvis, then you have a legitimate Googlebot.
so i figured out that there was NOT googlebot crawling my site. so i banned ip address. But if other user has the same ip address, he will not be able to access my site either?
What would be the best way to ban fake bots?
But if other user has the same ip address, he will not be able to access my site either?
What would be the best way to ban fake bots?
Right. If you are worried about this, you could ban bots by IP Address AND User Agent.
MSN has bots on hotjobs and hotmail IPs, as well as several other MS properties.
Savy cloakers do not use user agent cloaking -- instead they use IP address cloaking, or possibly IP address and user agent cloaking in combination. With IP address cloaking, the IP addresses of search engine spiders are kept in a database and the (IP address of the) requestor of a web page is compared to the IP addresses in the database. If a match is found, then the visitor is identified as a search engine spider, and optimized HTML is served.
