How does Google know about my hidden forum?

[viggen]

I saw today in my webmaster tools under site performance an example link to a topic in my discussion forums premium member section. There is no link to it publicly and you not only have to be logged on to see it, but also in a special member group.

So how does Google find out about those links and why does Google bother to sniff there around and give it as an example of site speed when its an error message anyway?

[graeme_p]

You may not have linked to it, but someone else may have - could one of your members have linked to it from somewhere?

[viggen]

@graeme, sure i cant control that and this might be a possibility althouth i doubt it, they are not very internet savvy users on my forum and its no use to link as you have to be special member group to see it anyway, and the question remains, why would Google make this page part of the site speed measurement. It doesnt make sense...

[koan]

Anyone using the Google toolbar or Chrome would help get your pages detected.

[indyank]

viggen, google has now gone beyond links to crawl content. Not sure how they are doing it and I am still trying to trace this. But I can confirm that from what I see on a few sites and some people are already aware of it.

[lucy24]

Analogy: Consider how many search results point to something interesting on page 17 of an article which turns out to live in a members-only database, so you have to pay to read beyond the abstract if you're not affiliated with a major university. I don't think it's because g### are in cahoots with the database owners. They've just never met a door they couldn't open.

[levo]

Definitely toolbar or Chrome.. I noticed my own internal text editing script among site speed samples.. It wasn't optimized for speed and it (probably) was dragging average speed down..

[robzilla]

GWT Site Performance statistics have nothing to do with links, crawling or indexing. They're drawn from usage data, pulled from various sources such as the Google Toolbar. Google isn't 'sniffing around' those private pages, but your users obviously are, and they might have the toolbar enabled, thereby sending (anonymous) statistics, such as page load time, to Google and consequently to your Webmaster Tools account.

[viggen]

@robzilla, thanks for that,

i just wonder why Google would show that particular link as an example in site speed, i have about 50 people that have access to that, so it is not even close to be relevant (i get about 10.000 unique visitors a day) so is that site speed data and the links that are shown in GWT just a random lucky draw of no relevance?

[tedster]

...when its an error message anyway?

What http status is served in the server header along with this error message?

[viggen]

@tedster
Result code: 404 (NotFound / Not Found)
I believe it is the standard error message from invision forum when a page is called that doesnt exist or one hasnt access to it...

[tedster]

If there's a 404 status in the http header, rather than just a text message in the page, then Google does not normally index the page. If they do, it's usually a short term error on their part.

[CovertSEO]

I was doing a search on a product on Google and interetingly the link to the download page was indexed by Google. If this was a newbie marketer I would understand the error but that was a seasoned marketer. Seems strange.

[DaStarBuG]

Let me guess, you have analytics on these pages as well ;)
Maybe even AdSense?

Then there are referals, Google Toolbar, Chrome.

So many ways that tell google about a webpage

[dstiles]

You missed out gmail, which I would not rule out.

On my Control Panel I block browsers that include the GTB string in their UA. It's easy enough to block others, including chrome itself, assuming the visitor is not savvy enough or corrupt enough to forge the UA.

[netmeg]

They show admin URLs for my various WordPress sites too, and there's no point in measuring those for speed. Nor can they even get into those pages as they are password protected. That's just Google.

[Sgt_Kickaxe]

Instead of writing out the list, which is extensive, on ways google gathers data let me give you an example of how NOT private the internet it.

Lets take this very post I'm writing as an example.

The instant I started writing this post Google knew my IP and the exact time I started writing it because of the javascript file that loaded with the little +1 button at the bottom of the page. Google will know the INSTANT I hit publish that mine was the first ever IP address on this page AFTER this is posted because of the +1 button javascript file that I will see on the next page after posting. Google will know that the IP they recorded with this forum posting is me the next time I log into my gmail or adsense since the connection will be made with my IP upon logging in. It's the same on all forums/blogs.

Even if there wasn't a +1 button down there most sites have SOMETHING Google. A search box, analytics, adsense, a 3rd party tracking system which shares data... the list is long.

[indyank]

Not all users have a fixed IP...In fact, majority of web users have dynamic IPs. If you say that they can use something like IP + machine identifiable info through those cookies, it might be valid to some extent.

But remember that google never gives away IP of visitors in its analytics tool and I think they even say that personally identifiable information are not gathered. This could be applicable to any of their products.

[edited by: indyank at 4:03 am (utc) on Jul 31, 2011]

[indyank]

netmeg, I am sure that you are one who wouldn't use a google toolbar or any of their products while accessing your wordpress sites or do you?

[dstiles]

Sgt - I have JS blocked in this forum so from that viewpoint they do not know who I am - I see no active JS on this page at all. I almost always block google JS nowadays anyway - froogle/base is the exception.

On the other hand, since I have had the same fixed IP for several years and have registered my domain with them for various things in the past (eg WMT), I am sure google knows exactly who I am IF they can grab my IP. They can probably make a shrewd guess just from my nickname here.

[Andem]

This is probably because of toolbar data. My browser homepage was mysite.com/myadminpage, without analytics and without adsense. Randomly it seemed, it was one of the top pages in G webmaster tools. The Google toolbar is the only way that Google could have known this page existed.

Since uninstalling the toolbar and basically ridding my system of anything Google and only using Chrome for testing purposes, the myadminpage that is my homepage has completely disappeared from my WMT account.

[justrobin]

Your forum member unknowingly have created an action that has created a certain. As you said, they are not internet savvy but we do not know what applications and programs have been used by these users.

Just a theory though...

[Bewenched]

I'm sure that just having a Google toolbar gives away alot more information about your surfing habits than they'll ever admit.