My site has been getting crawled by 66.249.65.208 and 66.249.65.229. Does anyone know if these are new bots or have any info about them?
tedster
8:05 pm on Feb 28, 2007 (gmt 0)
A reverse IP (also called reverse DNS) lookup says crawl-66-249-65-208.googlebot.com and crawl-66-249-65-229.googlebot.com. Also helpful is the list of googlebot IP addresses at [iplists.com...]
Are you seing anything unusual or new about the crawling behavior from those IPs?
tedster
7:26 am on Mar 1, 2007 (gmt 0)
Here's another thread that covers a nifty server-side approach to verifying Googlebot - something Matt Cutts showed us last fall. It works for slurp, too and iit can help keep out rogue and scraper bots spoofing as googlebot.
> Are you seeing anything unusual or new about the crawling behavior from those IPs? I am looking in my log files now but I suspect these bots are crawling an XML sitemap several times a day. There are other IP address doing the traditional spidering but these bots seem to be “load testing” the XML files.
