Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


levo - 11:12 pm on Sep 7, 2011 (gmt 0)


My code just caught 5 Google IPs. Request headers are;

User-agent: urlresolver
Host: www.domain.com
Accept-Encoding: gzip


and IPs are 74.125.42.80/2/3/4/5

Any idea what "urlresolver" is for? Something like facebook url linter?


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com