Google adbot is now hitting my site

I don't buy or sell or connect to ad networks?

[SumGuy]

My 17-year-old company website has started to be hit by the google adbot as of a few weeks ago. It's just 5 hits so far (4 of them to PDF files). I'm not concerned by the traffic caused by this bot (the regular google search bot hits us 50 to 100 times a day, grabbing the same files over and over again - you'd think they'd figure out we have a basically static site).

From what I understand about the adbot, it's trying to figure out something to do with a "landing page experience":

------
Understanding landing page experience
Landing page experience is Google Ads’ measure of how well your website gives people what they’re looking for when they click your ad.
-----------

Well, we don't have any ads, don't pay for any ads, and have no links at all to any ad networks, google-this or google-that.

I'm not concerned or want to block the adbot, I'm just curious as to why it's algorythms are pointing it to our site.

Could it be that someone else is linking to our site in a way that the google adbot finds interesting or relevant to it? I've noticed that Fecebook is hitting our site now in a very noticable way starting a month or two ago after years of basically nothing. We have no FB page or account, but have discovered that there was an autogenerated FB page for our company about a year ago.

[not2easy]

If you are not using Google Ads to refer traffic you may want to disallow that bot in your robots.txt file. It does obey.

I have blocked it for many years since I found someone figured out a way to use one of my pages as their "landing page" - from a completely empty WP site.

FB does auto-generate business pages which you can either claim or ignore. If you do not intend to actively cultivate the FB page, I would ignore it. It still can send traffic but I can't say whether it is human, productive traffic. I've seen some of them, they are just generic business listing pages with a link, phone, address, whatever is found in the serps.

[keyplyr]

Bots just crawl your pages... everyone's pages. In fact Most of Your Traffic is Not Human [webmasterworld.com]

The more you look through your server logs, the more you'll learn to understand what these bots do.

Also, please keep your posts specific to one issue so the discussion can be focussed and more helpful for others.

Utilise the Site Search (top-right) to find previous topic related discussions.

[justpassing]

Google Ads’ measure of how well your website gives people what they’re looking for when they click your ad.

This is why so many advertisers disappeared ... Good to know that this is not only publishers being evaluated ...

[SumGuy]

> The more you look through your server logs, the more you'll learn to understand what these bots do.

No, the understanding of what some of these bots are doing does not come from the logs. Knowing they are happening comes from the logs. Understanding comes from talking to people or stumbling across explicit explanations posted online (or following a url that could be present in the bot's user-agent).

I still don't know what the adbot's function is in terms of how my website is configured (ie - no advertising or click-tracking functionality or tie-in with any third-party or off-site links - I know all that sounds old-fashioned or quaint, but it works for us). I don't want to "make the google gods angry" by blocking / disallowing their adsbot access to my site unless it will absolutely have no effect on how we rank on google-search.

I also want to know if this new out-of-the-blue interest by adsbot could be because of how some possible unknown third-party is linking or referencing our site (in which case - disallowing the adsbot will interfere with the intent of this linking?)

[wilderness]

SumGuy,
There's a fairly recent thread on google ads request, that I believe keyplr started.
My primary site gets regular requests (not enough to be a pest) for ads.txt from Google and another SE (escapes me).
Generally speaking, disallowing any single page in robots.txt will not result in a negative effect for your site (s) overall.
Many have had the 80-95 Google ranges denied for a long time and that does not affect the normal google bots and/or rankings.

[not2easy]

The "Adsbot-Google" has the job of evaluating a landing page for a Google Ads campaign. The UA that requests your ads.txt file is Googlebot. If you do not have any Google Ads or affiliates that you allow to have Google Ads campaigns using your URLs there is no reason to allow the Adsbot-Google UA. Someone may be using your content to get better prices for their own Google Ads campaign or advertising "as you" to get your clients.

To keep them out, just add these lines to your robots.txt file:

User-agent: Adsbot-Google
Disallow: /

Since I don't run any Google Ads (used to be AdWords) campaigns, I block the Adsbot-Google on all my sites. They do obey, I have not seen them on any of my sites.

[keyplyr]

Personally, even if I didn't use Google Ads or Adsense, I would not disallow the Adsbot-Google. It doesn't cause any harm to your server.

It would be naive to think that bots, especially some of the more sophisticated bots, only check robots.txt to see if *they* are disallowed.

There are beneficial (to my interests) bots that may use the Adsbot-Google access as a metric. One such bot is the AhrefsBot, another is SemrushBot. There are often perifierial advantages that may not be initially apparent until researched.

[SumGuy]

I am *not bothered* by the google adsbot hitting my site!

I am bothered if someone else is doing something I'm not aware of in terms of linking or trying to monentize my site or site-content!

So again, my only question is -> in my case (when I don't believe the adsbot has any reason to hit my site) - does the adsbot hit my site *because* someone else is playing such games (ie monetizing or what-ever) with my site/content?

> AhrefsBot, another is SemrushBot

When I see those turn up in my logs I block the IP's they come from, and even the entire AS number they come from.

How can I find out if someone is playing games with my site/content? Will google control panel (or webmaster control panel or what-ever it's called) give me any clues?

[keyplyr]

Different bots do different things. Again, I invite you to do some reading here in this forum. Most all bots have been documented.

Linear bots just crawl. They do not need a reason. They follow links. If your web pages are indexed, they will get crawled.

Others bots are vertical. They follow some type of list. Link checkers are in this category.

Adsbot-Google is in both categories. While there *may* be a reason for Adsbot-Google to visit your web pages, IMO there probably is no reason other than your pages are on the internet.

I seriously doubt there's any conspiracy going on. Every site I work on has Adsbot-Google visits, lots of visits, whether they use ads or not. There is no reason to feel "someone is playing games with my site/content"

[not2easy]

> AhrefsBot, another is SemrushBot

When I see those turn up in my logs I block the IP's they come from, and even the entire AS number they come from.

If you wish to block specific bots, blocking the UA is much more efficient and doesn't require constant monitoring. It also avoids blocking legitimate visitors that can be in the same CIDR.

How can I find out if someone is playing games with my site/content? Will google control panel (or webmaster control panel or what-ever it's called) give me any clues?

No, you would need to be your own detective. I see no reason to allow that bot unless I am using their Ad services.

Do a search on Google for likely terms that might be used for the pages that you see the Adsbot requesting. You might see your own site URL or words from your site in an ad. You may never see an ad even if it exists because they may block your area from their campaign. It is unethical to click on paid ads just to see what happens, so I wouldn't just click around unless you do see your own content. If you're going to click on anything use a tool to record the headers. If you are thinking of reporting the ad, you should have proof. When that happened to me, it was well over a decade ago, I just right clicked on the ad that showed my content and "save file as > web page complete" that showed me the redirect header from the "empty" WP site.

To save yourself the time and trouble you can just Disallow the bot and let them find other people's content to run their campaign on. There may be no reason to be suspicious, but since my experience, I see no reason to allow it.

[keyplyr]

Well there are certainly a wide variety of website niches. Some bots are beneficial, some are malicious and some are benign depending on your site model. If they don't benefit your interests there may be no need to allow them.

My point was/is the Adsbot-Google doesn't cause any malicious activity. Even without ads on your site, allowing it is beneficial to getting backlinks and ranking due to other services using it as a metric.

When some bots request robots.txt they are checking for more than just their own UA directives.

When that happened to me, it was well over a decade ago...

Since then, there have been advances in technology that should stop things like that. If you're not already doing so, consider using all the applicable HTTPS Security Headers [webmasterworld.com]