Is Googlebot really rendering pages?

[JorgeV]

Hello-

I keep reading about Googlebot now only fetching the content of webpage, but also rendering the pages, using the chrome engine. However, I wonder if this is really true, and if there is evidence of it?

I mean, I perfectly understand that Googlbot can parse HTML, CSS and execute JS thanks to the Chrome engine, to render the page as it appears in a web browsers. But, even it can take a while to render a page. Even if Googlebot limits the rendering to , let say half a second, before stopping the rendering, multiplied by the number of pages that it crawls this is taking a very long time to render each page visited.

So I am wondering if there is evidence, that it renders pages, or may be a sample only? Or most important page, according to Google's criteria?

For example, if a text is within a DIV, at the end of the HTML code, but displayed at the top of the page because of CSS, does Google consider this text to be on top? (which "might" gives more value to this text).

What about div (or other elements), which are hidden on desktop, and visible on mobile, and vice versa? Does Googlebot only consider the text visible on mobile? (when mobile crawl is active). And, if a div's content is showing only after an action of the visitor (like "read more" to expand a block of text), what does Googlebot index?

[lucy24]

I hope that when you say “Googlebot” you mean “Google in all its manifestations”. The Googlebot as such just fetches. It doesn’t stop to look; that part is handed over to other parts of the vast complex of emulators and algorithms. At this late date--this is 2019, right?--I would say it's wildly improbable that any search engine, anywhere, looks only at the raw words in the html, without considering what it looks like to a human. After all, that's the whole point of “mobile-first indexing”: not just what’s there but what it looks like.

[tangor]

I suspect that G sees it all, and while pushing mobile, has not neglected desktop. Not really sure what the query is about ...

With all the resources g has at hand I'm pretty sure they see pages (as humans) regardless of device.

[JorgeV]

Hello-

I hope that when you say “Googlebot” you mean “Google in all its manifestations”. The Googlebot as such just fetches. It doesn’t stop to look; that part is handed over to other parts of the vast complex of emulators and algorithms.

Okay, sorry, if I misused words, I was just relying on the vocabulary used by Google itself:

Today, we are happy to announce that Googlebot now runs the latest Chromium rendering engine (74 at the time of this post) when rendering pages for Search
[webmasters.googleblog.com...]

[lucy24]

Yeah, it's not easy to keep things straight when G itself uses misleading terminology :)

But it's nice to know someone finally noticed that Chrome/41 is no longer the most recent version.

:: detour to look at linked article ::

OK, so “today” means 7 May, almost half a year ago. They certainly haven’t been in any hurry to update the crawler’s UA string--which on most sites would be irrelevant unless you’re serving different content to different version numbers, and who the heck bothers to do that? Otherwise they just take their findings home to Mama Computer, and then do the rendering.

:: further detour to raw logs ::

Nope, still not finding anything to match "Chrome/(?!41)" except a string of Android Googlebot requests back in April/May--right around the time they posted that article--when someone clicked the wrong button so the UA (from crawl ranges) came through as

Mozilla/5.0 (Linux; Android 9.0.0; en-us; Pixel 3 XL Build/PD1A.180621.003) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko) Chrome/[CHROME_VERSION] Mobile Safari/[WEBKIT_VERSION] (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

(Really.) Funny that I can't remember noticing at the time. It’s so goofy, it would have merited an SSID post.

[NickMNS]

Google crawls in one step, and then renders at a later time.

Here is a video from Google I/O 2018 that explains how it all works and what you need to know.
[youtube.com...]

[aristotle]

Well the Chrome browser fetches it and then renders it on the screen for human eyes.

The ranking algorithm probably doesn't "visualize" it the same way a human does.

[lucy24]

Well the Chrome browser fetches it

Except that there is no Chrome browser; it’s just a database sending a user-agent string to go with its request.

Imagine a different time, a different place, where there are no robots and all crawling is done by actual humans in {pick a poor country at random} sitting in front of actual computers running actual browsers yielding content that they then view with their actual eyeballs. In that scenario, the browsers probably would be extremely elderly--Chrome/41, say--that being all the humans could afford. But not in this time, this place.

:: wandering off to pursue mental picture of steampunk internet ::

[not2easy]

mental picture of steampunk internet

A Brazilian Guide to the Universe?

[NickMNS]

Except that there is no Chrome browser; it’s just a database sending a user-agent string to go with its request.

I think it is worth pointing out that the html delivered by the server at the time of the request can look significantly to the final product, that is once the dom is fully loaded and rendered.

[lucy24]

can look significantly to the final product

¿ significantly different to ?

And of course it's not just the html; there's a reason the search engines want to see all your css and scripts, because that's what will determine what the human user ends up seeing. (To take the most blatant example, if a part of the page is styled as {display: none} or {color: transparent} or {font-size: 500%}, that's something the search engine needs to know.) And on top of that there's a reason why the major search engines always name some specific page as referer when requesting scripts and styles, as they've done for several years now. They want to be sure they're getting the same css that a human would get when viewing that page.