Sorry I've been away for a bit, folk, but has anyone seen SEMRush posing as Googlebot?
Today I got this in Wordfence: IP: 46.229.173.66 [block] Hostname: siteaudit.crawler.semrush.com Browser: undefined Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
keyplyr
2:32 am on Nov 21, 2017 (gmt 0)
Welcome back oddsod,
SEMRush's siteaudit.crawler may be testing to see how your site reacts to Googlebot. Do you have an account at semrush.com? If not, someone else could be auditing your site, possibly a competitor.
Many of us block all Googlebot UAs that do not come from authorized google crawl ranges: [webmasters.googleblog.com...]
Googlebot is arguably the most faked SE UA.
oddsod
12:49 pm on Nov 24, 2017 (gmt 0)
Many thanks for the reply, keyplyr.
I don't have an account at semrush. In fact, I'm not keen on all these SEO bots (perhaps for the reason you mention - competitor monitoring).
But it looks like I'm going to have to block SEMRush on their IP/hostname now if they are faking being Googlebot.
lucy24
6:36 pm on Nov 24, 2017 (gmt 0)
it looks like I'm going to have to block SEMRush on their IP/hostname now if they are faking being Googlebot.
You could, but in general it's more useful to follow keyplyr's approach of blocking anything that calls itself Googlebot but doesn't come from a known Google IP. (You don't need to do a DNS lookup every time. Just say 66.249.whatever-it-is.)
:: detour to raw logs ::
Huh. Is 192.243.blahblah really Semrush, or a faker? I kinda think a faker, since they're getting blocked while the 46.229 aren't.
