Is there a UA setting that allows all Google Bots?

[born2run]

Hi so I need the common UA for Google's bots that are used for Search Engine Console (Life URL test), and the G Mobile friendly test to check a website resources loading.

For now I've allowed AS15169 to allow all Google bots but it opens up a flood of bots from Google Cloud servers too.

Is there a setting I can set on Cloudflare to look for specific User Agent's string for all Google Bots related to SE Console & Mobile friendly test? Thanks!

[born2run]

Hmm so i did some search for Google bots UA and it seems all of them contain this string:

+http://www.google.com/bot.html)

Can I just set a Cloudflare FW rule to allow UA that contains the string above? Thanks!

[not2easy]

Users pretending to be googlebot use the same UA string as the real thing. Google has a page to tell you what exact UAs they use. Co-incidentally it is that same URL that you have posted as part of the UA - without the + of course.

[lammert]

Thanks for letting me in. I use that UA string also for my malicious bots :)

You should look into the Cloudflare cf.client.bot firewall rules instead. They are specifically designed to allow friendly bots, while blocking the imposters.

[born2run]

Lammert what number rule are these?

[born2run]

OK thanks lammer! I got the rule (Field Known bots - On- Allow) in CF, and things are working better now! I've disabled AS15169 rule allowing as fake bots from G Cloud began visiting my website.

Now the other thing is my google analytics code is being messed up (not loading). Could this rule also fix this GA js code problem? Thanks again!

[lammert]

You should use it in a Firewall expression

(http.user_agent contains "bot" and http.user_agent contains "Google" and not cf.client.bot)

This rule blocks bots using "Google" and "bot" in the UA, but are not in the Cloudflare approved list of bots.
This was an answer to your first question. I don't know if this will fix GA issues

[not2easy]

@ born2run - Is this the same site you are asking about in your GA thread about where to add the GA code? [webmasterworld.com...]

It helps to keep different discussions separate so others don't get confused. ;)

[wilderness]

^66\.249\.(6[4-9]|[7][0-9])\.
are the ranges for google bots.

[not2easy]

@wilderness - that is true for the standard Google Bots but they use some alternate IPs for some of their tools. I needed to unblock 66.102.0.0/20 and 66.102.128.0/20 for verification bots. I have also noted their GSC tools do not always crawl from standard ranges for tests like URL Inspection and Mobile Friendly evaluations. The only way I see they are blocked is in error logs and in raw access logs.

[NickMNS]

Now the other thing is my google analytics code is being messed up (not loading). Could this rule also fix this GA js code problem?

No! Stop... Google Analytics never sees your server, Google Analytics is loaded client side, there is nothing you can do with UA or anything else on your server that can fix problems with GA.

You are mixing up issues here and you have three separate threads trying to resolve it:
[webmasterworld.com...]
[webmasterworld.com...]

I must assume that the issue you are trying to fix is your issue with mobile friendly test. Is this correct?

If it is please confirm and then maybe the mods can redirect this thread to that one.

If not then describe what the problem is exactly?

Note that the premise of the title of this thread is flawed from the get go. There should be no reason to "Allow" Google via the UA, implicitly this suggests that some how your a blocking all UA's and then only allowing a subset through, given that it is impossible to know the full universe of legitimate UA's this approach seems flawed. I could be wrong here, as I this not my area of expertise.

[born2run]

Yes @not2easy same site multiple issues.

[born2run]

Yes actually it’s this google mobile friendly bot test failing to load all my sites css jpg etc files. It affects my seo ranking too as I see same issue with Google search console live url test. My apologies.