Google tools DO result in googlebot visits

[notsleepy]

I know there has been previous talk about whether or not Google toolbar queries result in pages being spidered. I remember GoogleGuy absolutely insisting that this was not the case.

This blog doesn't refer to the toolbar but Google tools built into Opera. Interesting nonetheless:

[sidhe.org...]

I like Dan's last test:

I put up a brand-spanking-new image, completely unlinked to anything. Downloaded Opera for Linux, fired it up, told it that I was OK with google, and looked at the new URL. Then I shut down Opera.

5 minutes 38 seconds later... here comes googlebot!

[rfgdxm1]

One comment here. From what I recall, GG posted that quite some time ago. Thus, what he wrote may have been accurate *then*. I don't recall him saying it was guaranteed to be this way for all time.

[georgeek]

Extrapolating from a single experience is unsound. Assume it was a coincidence until there are multiple and reproducible results.

[rfgdxm1]

Logic georgeek. He posted that he uploaded an image unlinked from anywhere else. He then accessed it with his Opera. Less than 6 minutes later Googlebot showed up. How else could Googlebot know about it?

[georgeek]

rfgdxm1 Without forensic analysis of the details it is impossible to say what really happened. Google crawling IRC for example but until tool bar users can reproduce this behaviour, in well documented circumstances, it remains conjecture. Logic tells us not to believe all that we read :)

I have ran several experiments with the IE toolbar when this was first reported - all with negative results. I shall run them again and report back if there has been a change.

[rfgdxm1]

While I will remain skeptical without adequate reproduceability, if the person who wrote that tells the truth about what he observered, then it is reasonable proof for *him*, because there is no other logical explanation. As for using the Google Toolbar, note this person used Opera on Linux. It is possible that Google currently is doing this only with these Opera users.

If Google isn't doing this now, I'd expect it in the future. With those "Supplemental Results", they already are spidering orphan pages they had found before.

[ukgimp]

[webmasterworld.com...]

[Dave_Hawley]

Same thing *could* have happened if he said "Google" backwards three times.

Dave

[edited by: Woz at 2:36 pm (utc) on Nov. 21, 2003]
[edit reason] No URLs please, see TOS#13 [/edit]

[valeyard]

Opera displays Adwords ads, to choose these Google needs to know what the page is about. It's not Opera per se that's calling the bot, it's Adwords.

I'm in Adsense and I know from my logs that if I put up a brand new page with Google Ads and look at it via any browser then Googlebot will come running very soon. Every time.

BUT, before you get too excited (like I did)...

This is a special visit with a special tag (Mediapartners-Google). The page visited does NOT seem to make it into the main index until a standard Googlebot crawl arrives, whenever that might be.

Maybe this is what the supplemental index thing is about?

[yonnermark]

Same thing *could* have happened if he said "Google" backwards three times

El Goog
Is that what they call it in spain?

[Chndru]

Good point, valeyard. The moot Q is did he had the Google Ads option checked in Opera or just the regular banner. If it were the Google Ads, it has to spider the webpage, to see what Ads would be relevant there.

[DanSugalski]

I had the Opera defaults set. I'll be honest and say I don't know what they were, nor did I really care, as I was looking at the question "Does opera, with the default google settings, cause a visit from one of the googlebots?" Which it did. This was a follow-up to the question "Was there a hidden googlebot or google irc server on the perl.org IRC net?" Which apparently it does not.

Note that I didn't care which googlebot, nor did I care (other than in a vague academic way) whether the image ended up in google's directory. I just wanted to know whether opera would trigger a visit from google, and it does.

While I've not done any particular testing, the google toolbar doesn't seem to trigger the same sort of fast bot activity. Whether it ultimately triggers any is sonething I didn't look into. It might, or it might not, but I'm not interested enough in the results to go check.

[GoogleGuy]

"The page visited does NOT seem to make it into the main index until a standard Googlebot crawl arrives, whenever that might be.

Maybe this is what the supplemental index thing is about?"

valeyard's pretty much said it all, except for the last question (relation between the adsense/mediapartners bot and the supplemental results). The answer to that is pretty clear--there is no shortcut into our index via advertising or adsense. Being an adsense publisher won't help you get any extra pages into Google's search index.

Chndru also put it pretty well.

[Splosh]

I downloaded Opera Browser for windows and allowed it to communicate with google.

I then navigated to a brand new site, then checked logs and found a hit from "Mediapartners-Google/2.1"

When you allow opera to Comunicate with google, you do so to allow Google to place text ads (AdWords) at the top of the browser window. Opera generate income from anyone that clicks on these ads through the AdSense program. Mediapartners-Google/2.1 is the AdSense spider used to check a web page and then return relevant ads. You should notice the Ads change each time you visit a new page.

[shasan]

Ok, I just looked at my stats for the last month and found that

crawler8/9.googlebot.com:Mediapartners-Google/2.1 :http://www.googlebot.com/bot.html

is the name of the bot that has been doing the extra-curricular crawling on my site.

I'm assuming this is the Mediapartners bot.

I have it accessing pages with no incoming links (like mysite.com/controlpanel/admin, and banners/admin) and 'nowhere near' a google adsense instance. Why would it want to crawl the authentication page for my link tracker utility?

The only thing remotely relating those pages to google is the fact that i have used the Google toolbar when visiting those pages.

There's too much of this happening in my logs for it to be a coincidence. There's no other way for these pages to have been known. is there?

curiouser and curiouser.