Welcome to WebmasterWorld Guest from 3.80.38.5

Forum Moderators: Ocean10000

Message Too Old, No Replies

SEMRush posing as Googlebot?

     
2:47 pm on Nov 20, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2005
posts: 2264
votes: 0


Sorry I've been away for a bit, folk, but has anyone seen SEMRush posing as Googlebot?

Today I got this in Wordfence:
IP: 46.229.173.66 [block] Hostname: siteaudit.crawler.semrush.com
Browser: undefined
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2:32 am on Nov 21, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


Welcome back oddsod,

SEMRush's siteaudit.crawler may be testing to see how your site reacts to Googlebot. Do you have an account at semrush.com? If not, someone else could be auditing your site, possibly a competitor.

Many of us block all Googlebot UAs that do not come from authorized google crawl ranges: [webmasters.googleblog.com...]

Googlebot is arguably the most faked SE UA.
12:49 pm on Nov 24, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2005
posts: 2264
votes: 0


Many thanks for the reply, keyplyr.

I don't have an account at semrush. In fact, I'm not keen on all these SEO bots (perhaps for the reason you mention - competitor monitoring).

But it looks like I'm going to have to block SEMRush on their IP/hostname now if they are faking being Googlebot.
6:36 pm on Nov 24, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15444
votes: 739


it looks like I'm going to have to block SEMRush on their IP/hostname now if they are faking being Googlebot.

You could, but in general it's more useful to follow keyplyr's approach of blocking anything that calls itself Googlebot but doesn't come from a known Google IP. (You don't need to do a DNS lookup every time. Just say 66.249.whatever-it-is.)

:: detour to raw logs ::

Huh. Is 192.243.blahblah really Semrush, or a faker? I kinda think a faker, since they're getting blocked while the 46.229 aren't.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members