Toolbar a threat to private Web pages?

Does it lead to indexing URL's that are not linked from anywhere?

[ThatAdamGuy]

Hi there,

I heard a rumor that the Google Toolbar -- with advanced features turned on -- would "phone home" to Google and invite one of the GoogleBots to index any 'new' page discovered.

Is this true? Meaning, if I surf to a page that's publicly accessible BUT not linked from anywhere, will it suddenly get indexed?

I ask this because I just installed a new message board on my site, and one of the features of this board is that it can display the GoogleBot as a visiting member in the "Who's Online" section based upon IP detection.

Within minutes of making a post, BAM, there's the GoogleBot, reading that post! (I spied the GoogleBot also in the "Personal Messenger" section one time, too, but alas, never heard from him :D).

I do have AdSense ads on my forum (since there are some pretty highly-delineated topics, such as travel, college life, etc.), so maybe it's the AdSense Google Bot visiting? Can I tell by IP address?

I know, I know, a lot of questions :D

[takagi]

I'll say it again. I don't think our privacy policy prevents Google from doing this, because we are allowed to use anonymous user data to improve our search, but installing the toolbar didn't make googlebot crawl your page. See
[google.com...]
for some of the typical ways that urls leak. Other ways include people guessing urls, network/DNS setups, etc.

GoogleGuy at msg 13 in the thread: GoogleBot visits what you visit if you have the toolbar [webmasterworld.com]

[ThatAdamGuy]

Okay, I've now checked this three times.

1) Make a new post on my message forum.
2) WITHIN TEN MINUTES GoogleBot is on that exact topic.

Specifically, the IP is 64.68.87.41 -- crawler8.googlebot.com.

Can anyone offer an explanation to me that DOESN'T basically point to the Toolbar leaking this URL?

Don't get me wrong. I largely agree with GG's comments about security-through-obscurity=bad. And overall, I'm pretty happy to have every page in my forum indexed within minutes after its written :D

But I do think there's some cause for concern here with regards to Google's disclosure on this issue. I know GG is not an official spokeman, but unless I'm really missing something, this quote is pretty far off the mark:

[...]but installing the toolbar didn't make googlebot crawl your page.

I look forward to hearing other plausible theories :)

Regards,
Adam

P.S. -- Even funkier, and complicating matters... HotBot showed up to check my latest forum messages just 45 minutes after GoogleBot. Is my innocuous forum on some sort of CIA watch list?!? :D

Keep in mind that I've had this new forum up for precisely 2 days. I didn't even link to the forum from other site pages until a few hours ago.

[ThatAdamGuy]

Now Google was even faster!

Posting time: 4:22am
Crawler9 arrived: 4:25am
Amount of sleep Adam is gonna get after staying up this late: Very little.

Oh, and in case people are curious:
- I'm using the new 1.2rc1 version of InvisionPowerBoard for my forum software. Highly recommended freeware, btw!
- The board is located at www.mysite.com/forums/. This is NOT the same URL as my old board!
- There is very little content on the new boards at this point (understandably).
- Each time GoogleBot is prompted to visit my forum after someone posts, it also "clicks" on other random pages of my forum. The funniest time was when the online list showed "Gary the Googlebot: Using Personal Messenger." Okay, maybe you just had to be there. :P
- I'm using the Toolbar 2.0beta on myIE2 with Win XP Home.
- My favorite color is green.

[edited by: engine at 11:46 am (utc) on July 3, 2003]
[edit reason] de-linked [/edit]

[ThatAdamGuy]

Oops, let me clarify.

The crawlerbot crawled the exact page I had just newly posted three minutes later. It wasn't an instance of it just showing up on a random page ;)

And interestingly enough, these are dynamic pages, in the form of:

http://www.mysite.com/forums/index.php?showforum=5

I didn't even think Google liked those sort of pages to begin with.

Okay, now I'll shut up and go to sleep :) Dang geek tendencies ;)

[edited by: engine at 11:47 am (utc) on July 3, 2003]
[edit reason] de-linked [/edit]

[Chris_R]

The argument to me is it is a waste of resources - and doesn't make sense. Google isn't going to rank you high - unless you have external links - so why go to sites that might not have them. If anything - they would be better off crawling domains as they are added to the zone.

Some people - including one I think knows what they are doing pretty much swears that you are correct in your suspicions. I just don't think it makes sense.

[DavidT]

The index page of a test site I am working on has a PR of 0/white bar, all internal pages greyed out. I block the entire Internet from it except my own fixed IP. How could it have a white bar?

[cdkrg]

It's because of AdSense.

[srinivas]

how much time we have to stay at the site for the toolbar to pick up the page