Google 'pre-requesting' top URL in SERPs?

Page 1 of SERPs making 'GET' request for Position 1 URL

[Winooski]

Do you ever use the Firefox Live HTTP Headers extension [livehttpheaders.mozdev.org] to look at the server requests for the first page of the Google SERPs? I'm noticing what appears to be a very strange phenomenon: Google is requesting the URL for the topmost organic result, even though I haven't clicked on the result.

Here's an example for [ibm]:

http://www.google.com/search?hl=en&q=ibm&btnG=Google+Search[br][br]GET /search?hl=en&q=ibm&btnG=Google+Search HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.google.com/webhp?hl=en
Cookie: PREF=ID=11bc411c013cae23:TM=1190058153:LM=1190058153:S=ktpKPSerfu9xUusp[br][br]
HTTP/1.x 200 OK
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Server: gws
Transfer-Encoding: chunked
Content-Encoding: gzip
Date: Tue, 18 Sep 2007 16:15:36 GMT
----------------------------------------------------------
http://www.ibm.com/[br][br]
GET / HTTP/1.1
Host: www.ibm.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.google.com/search?hl=en&q=ibm&btnG=Google+Search
X-Moz: prefetch[br][br]
HTTP/1.x 302 Found
Date: Tue, 18 Sep 2007 16:15:36 GMT
Server: IBM_HTTP_Server
Location: http://www.ibm.com/us/
Content-Length: 206
Keep-Alive: timeout=10, max=96
Connection: Keep-Alive
Content-Type: text/html
----------------------------------------------------------
http://www.ibm.com/us/[br][br]
GET /us/ HTTP/1.1
Host: www.ibm.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.google.com/search?hl=en&q=ibm&btnG=Google+Search
X-Moz: prefetch[br][br]
HTTP/1.x 200 OK
Date: Tue, 18 Sep 2007 16:15:36 GMT
Server: IBM_HTTP_Server
Cache-Control: no-cache
Vary: *, User-Agent, Accept-Encoding
Keep-Alive: timeout=10, max=86
Connection: Keep-Alive
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
X-Pad: avoid browser bug
Content-Encoding: gzip
Content-Length: 6884
----------------------------------------------------------

Again, I am *not* clicking on the link to IBM; Google appears to be doing it on its own.

You should be able to verify it yourself using Live HTTP Headers. Here are some other queries that yield the same thing:

[google.com...]

[google.com...]

[google.com...]

In my tests, there were definitely queries ("long tail"?) that did not yield the "pre-requested" top-ranked URL, but I'd guess 80% of the queries I tried did.

I'm wondering if anyone else can confirm this, and whether it's really a Google issue or perhaps an issue with one of my Firefox extensions. Last night I asked a colleague to try the experiment from his home PC, and he confirmed the same behavior. (My apologies if this has already been documented and explained. I wasn't able to find any mention of it anywhee.)

[CernyM]

I think this is a Firefox pre-fetch thing rather than a Google thing.

[Winooski]

Thanks CernyM. However, I should add that I did not see the same thing with either Yahoo! Search and Live Search. (Also, why would Firefox inconsistently fetch just the #1 organic result URL?)

I don't have the Google Toolbar installed, either...

[jimbeetle]

The gray matter is a bit fuzzy today, but I seem to remember Google doing this quite a long while back, if not for all searches, at least some that met some sort of criteria.

[tedster]

There's been a lot of past discussion here about the "Google pre-fetch" and "Google Web Accelerator" that you can find using our Site Search [webmasterworld.com]. Here are two of the more in depth discussions:

Google Enables FireFox Prefetching [webmasterworld.com]

Google Windows Web Accelerator [webmasterworld.com]

[youfoundjake]

Could it be part of the I'm feeling lucky?

[jimbeetle]

Thanks, Ted, that's what I was trying to remember. Googleguy from tedster's Firefox prefectch link:

Note that even if your show up at #1, this tag still won't be shown most of the time. It's only shown if we're pretty sure that the user will click on the top result. So if webmasterworld.com is #1 for the search "webmaster world" then we might add the prefetch tag. But we wouldn't if WebmasterWorld were #1 for another search. It can actually make your site look like it loads faster; I'd give it a week before you assumed that it created new load.

[Winooski]

Thanks Tedster! Without knowing that magic word "prefetching" (or even "Web Accelerator"), all my attempts to search about this were coming up empty.

So my takeaway from all this is:

(1) It's a default opt-in setting in Firefox that I can turn off by browsing to "about:config", then scrolling down to network.prefetch-next and double-clicking it to toggle it to "false". (Aside: "Whooo-weeee, lookit all them fields I kin change!")

(2) Though this default prefetching doubtless messes with some webmasters' pageview and session stats (especially if they're relying on their server logs), it probably doesn't skew things too badly...unless you're the webmaster for, say, Apple, and you're wondering why the homepage at www.apple.com/itunes/ has such a large abandon rate.

(3) I can shake my tiny fist in rage at these would-be Google-and-Firefox shenanigans, or I can just let it go.

I guess I can let it go. Sigh. Thanks again!

[g1smd]

If you have a #1 result in Google, your "logged traffic" will be higher than the number of eyeballs that actually saw your site because of this.

Some people will not actually be viewing your site, even though the browser already pre-fetched it.