Googlebot and Googlebot-smartphone... separate crawlers?

[vphoner]

I was reading an article about mobile first, and they claimed that there are two robots from google. One for desktop and one for mobile called Googlebot-smartphone. I only see Googlebot /2.1 in my log files. No sign of any mobile Googlebot. Any idea what is going on? Do you need to submit something to google to tell me to index mobile pages?

I have not gotten a notice from Google about Mobile first yet. In the article I read it seemed to indicate that even before Mobile First, there should be a mobile googlebot indexing your pages.

Here is the article I referenced: [botify.com...]

[keyplyr]

there are two robots from google

There's also the Mediapartners-Google bot that crawls for images, video and Adsense.

Is your site Mobile Friendly? [search.google.com] Or do you only have a Desktop version of your site? If you haven't seen the mobile Googlebot, don't worry about it.

Do you need to submit something to google to tell me to index mobile pages?

No, but keeping an updated sitemap.xml in your Google Search Console account [google.com] is helpful.

Each site will get crawled for the Mobile-First Index [webmasterworld.com] one by one. You will receive an email from Google when that has been accomplished. There's nothing you can do to speed up the process.

[lucy24]

One for desktop and one for mobile called Googlebot-smartphone

How old is this article?! Years ago, there were an assortment of UAs containing the element Googlebot-Mobile. The oldest ones--Samsung and DoCoMo--limped along until late 2016, but by then their primary mobile crawling was with an iPhone UA. But already in February 2014 they had dropped the “-Mobile” component of the iPhone UA string. Around April 2016 they changed over to an Android UA.

I can say this with great precision because I am currently, as we speak (I mean literally: I've got windows open) updating my “At Home with the Robots” summary.

[keyplyr]

How old is this article?!

15th December 2017

[lucy24]

And they think there's a “Googlebot-smartphone” UA by that name?

:: detour to article linked above, followed by further detour to its referenced source [seroundtable.com] ::

Yup, that's what I thought. What John Mueller actually said--assuming seroundtable quoted him accurately, while botify did not--was (emphasis mine):

if you look now probably something like 80% of the crawling is with the Googlebot desktop and maybe 20% is with mobile with the smartphone Googlebot.

Otherwise known as the Googlebot Android UA, which has existed for years.

Moral: It isn't enough to check your sources. You also need to check your sources' sources. Or, hey, just look at your logs.

[keyplyr]

It isn't enough to check your sources. You also need to check your sources' sources. Or, hey, just look at your logs.

Indubitably

There's also Googlebot-Image/1.0 which just yesterday grabbed 9k of my images from one site.

[vphoner]

keyplyr +, I only recently went mobile friendly (First week in June). In google search console, I saw that 95% of my pages are now listed as mobile friendly. I would assume that some sort of bot came and indexed for that. But I have not gotten the notice for mobile first. With all the penalties for not being mobile friendly (even on desktop pages), I would have thought that I would see a boost in traffic. I guess the question is how long does it take once you go mobile friendly for google to give you that boost (from not having penalties). Can it take months to fully kick in?

[keyplyr]

Be patient. It takes a while. Everyone is doing it :)

The ranking boost is very subtle & may not be noticed. It's almost futile to see your own ranking anyway since the AI algo now uses user intent instead of keywords to rank site and each search is customized using many metrics.

[vphoner]

Really hard these days. I took a big hit on April 15th, and another in Mid May. Organic traffic was cut in half. Not sure what the algorithm was in this time period, but I am hoping it was related to mobile friendliness. Still ranking #1 on some keywords, but many have gone from page 1 to page 4 or 5. Very frustrating not knowing what caused this...... So mobile friendly was my main weakness, and finally fixed that in June. Hoping...but if its something else, need to fix it. I also just started a Twitter account which I have read is important for ranking. Already had Youtube and Facebook accounts. But like many of us, flying in the dark and trying to fix the unknown.