homepage Welcome to WebmasterWorld Guest from 54.163.91.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
strange googlebot referer
googlebot referral from odd site
dupres01




msg:4692666
 3:39 pm on Aug 3, 2014 (gmt 0)

66.249.79.3 - - [02/Aug/2014:18:51:55 -0600] "GET / HTTP/1.1" 200 11420 "http://fr.wschange.com/new/2013/09/12/wildplum.biz/redirect" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"


never seen this before. is there some relationship between google and fr.wschange.com?

 

Pfui




msg:4692696
 7:35 pm on Aug 3, 2014 (gmt 0)

It's rare to see detailed G referrers nowadays and I don't know how to explain them when I do. Back in May I made a note of seeing --

crawl-66-249-66-14.googlebot.com
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

-- with a StumbleUpon ref:

http://www.stumbleupon.com/su/123ABC/www.mysite.com/dir/filename.html

Beats me. I just figure oddities happen.

lucy24




msg:4692744
 10:18 pm on Aug 3, 2014 (gmt 0)

It's rare to see detailed G referrers nowadays

For page files, that is. For some types of non-page files, notably js, a referer is very common. But the referer will then be your own page.

:: detour to check ::

Nope, can't find any at all. (Real googlebot, I mean. There's some Ukrainian referer spam, even if I constrain the search to 200 responses.)

Key_Master




msg:4698189
 2:50 am on Aug 27, 2014 (gmt 0)

I see googlebot leaving referrers quite frequently, sometimes using a googlebot user agent and other times using browser agents. Here's one that hit twice on the 11th. Each time a different referrer was used:

46.165.231.151
crawl-46-165-231-151.googlebot.com
Mon Aug 11 22:46:22 2014
HTTP_ACCEPT{'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'}
HTTP_ACCEPT_ENCODING{'gzip, deflate'}
HTTP_ACCEPT_LANGUAGE{'ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3'}
HTTP_CONNECTION{'close'}
HTTP_HOST{'www.example.com'}
HTTP_REFERER{'http://www.banffspringsgolf.com/'}
HTTP_USER_AGENT{'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0'}

That might be an unknown googlebot ip range to WebmasterWorld users.

not2easy




msg:4698190
 3:36 am on Aug 27, 2014 (gmt 0)

Um, that's because WHOIS says it belongs to Leaseweb/NETDIRECT.

Key_Master




msg:4698205
 4:46 am on Aug 27, 2014 (gmt 0)

That is not unusual. Google leases ip allocations from a number of internet service providers located all around the world.

keyplyr




msg:4698210
 6:16 am on Aug 27, 2014 (gmt 0)


That is not unusual. Google leases ip allocations from a number of internet service providers located all around the world.

Well I suppose there is the remote possibility that is accurate... I don't work at Google so I don't know for sure :)

However the most likely scenario (and one I feel *is* accurate) is the hit from the Leaseweb Server Farm is a malicious actor spoofing as Googlebot. I get about 20 to 50 of these spoof attempts from Leaseweb per month (and thousands from other server farms and infected machines) and I say "attempts" because I block them all.

MickeyRoush




msg:4698212
 7:05 am on Aug 27, 2014 (gmt 0)

How to verify Googlebot:

[googlewebmastercentral.blogspot.com...]

[edited by: MickeyRoush at 7:06 am (utc) on Aug 27, 2014]

bhukkel




msg:4698213
 7:06 am on Aug 27, 2014 (gmt 0)

Never trust the PTR record of IP address. It can be fake.

So double check: IP -> PTR record and then PTR -> IP address.

crawl-46-165-231-151.googlebot.com is not resolving to 46.165.231.151 so it is fake.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved