Forum Moderators: open
If so, it seems that people may unknowingly reveal pages that are intended to be private. While "security by obscurity is no security", the unintended consequences could be bad.
I have the same question about the Alexa toolbar, if anyone knows the answer to that too.
I didn't have a good answer. So, anyone know the real deal?
Welcome to WMW Steve!
I have seen instances of small, new sites with absolutely no links being found in the Google index - after I've viewed them using IE w/toolbar
I seriously doubt google would do this. Several pages I visit have passwords in them such as:
www.example.com/aws/432/password/
I know this isn't a good method of security, but I have no choice - as those are the way those companies operate.
There is no reason for google to visit sites you are visiting with the toolbar. Google grades sites based on the number and quality of links to it.
Why would they waste resources going to these sites - when they could just follow links off the web? They won't list sites that don't have at least one link to it anyway.
I don't go along with the whole obscurity thing, though. There are just too many ways for URLs to leak.
Google's privacy policy (http://toolbar.google.com/privacy.html) says it grabs the URLs (obviously). It doesn't say that it uses them for spidering, but it doesn't specifically say that it doesn't.
Alexa is more specific (http://www.alexa.com/help/webmasters/index.html), and I'm adding a robot exclusion rule for them. (I don't necessarly want to add robot exclusion for all search engines).
What else it going use them for? Google print them out and paper the walls with them or something? Some members of the board have a theory that the toolbar data can be used to ID SEO types, by finding unusual activity patterns
It would make NO SENSE.
I put up new sites a few times a week. There are plenty of people that come there with no links to it.
If you haven't gotten an email from someone spamming you to help you place your new site in the search engines - then you aren't making enought sites :)
MAYBE it is some sort of test thing. It would make no sense, and would be dangerous, for them to add pages this way. I could see maybe visiting the root page for some sort of test thing.
This would be a huge waste of resources. I would be amazed if this were the case.
Nobody ever knew about this (except for the hosting company with a lot of customers, so they won't ask google to index that), there's no inbound links to that site and the site is not listed anywhere. Google's last update shows that in the search results (which would be considered as SPAM I guess, since the sites are exact duplicates)..
A page's ranking (which is what I suspect you meant anyway) may very well should have extra emphasis put on it by the amount of trafffic it receives.
Google did mention this in one of their papers.
However, google is smart enough to know the difference between one webmaster working on their pages and legitimate traffic. They would most likely only count each toolbar once.
This will probably be used to some extent in the future - especially after google starts customizing their pages for specific users.
I am sure you are right that google is collecting this data, I don't think they are using it yet.
And your point about PR is well taken in that a page with good page rank doesn't mean it gets many visits - such as the yahoo privacy pages. Great PageRank, but who cares about those pages.
the toolbar I am sure holds great promise for the future.
Why is Googlebot downloading information from our "secret" web server?
It is almost impossible to keep a web server secret by not publishing any links to it. As soon as someone follows a link from your "secret" server to another web server, it is likely that your "secret" URL is in the referer tag, and it can be stored and possibly published by the other web server in its referer log. So, if there is a link to your "secret" web server or page on the web anywhere, it is likely that Googlebot and other "web crawlers" will find it.
THEY DON'T WANT PAGES THAT DON'T HAVE LINKS TO THEM.It would make NO SENSE.
Actually, that makes perfect sence they want to find all pages on the Internet. Think of the Internet as a big web, well sometimes it is multiple webs and one web does not touch another web. This is a way for them to find non-touching webs. It is also a way to locate content from a niave or lazy webmaster. They are tilling the earth and they will leave no stone unturned.
I know for a fact that if you visit a page with GGbar that GGbot will visit it too!
There is a simple test. Make a webpage, place no links to it. Visit while viewing with GGbar, wait 3 months. And check if your page is in. I did this. And guess what! That page is now in Google.
Sorry, chris_f, your url leaked out some other way. :)
Thanks, GoogleGuy. That's exactly the conclusive answer I wanted. I read your policy as not saying anything about this (I don't consider a plain URL to be personally identifiable info), so your statement is what I need. Would you consider adding that to your stated privacy policy?
And as Googleguy pointed out, the privacy statements almost always include language that makes it O.K. for them to implement such a policy in the future.
Having had Alexa show up and begin tearing through a not-for-the-public site in the past, I've made it a standard policy to not use any toolbar equipped browser when doing any kind of sensitive work.
Even though I believe that Google isn't currently using their toolbar for crawling purposes, I'm also not real confident that they'll send me a personal email If/when they change their mind. :)
To add to the discussion though and to see if any new info has come to light, It would not surprise me if google added urls visited to their database to crawl and if it already exists then they skip it and if it doesn't then it gets crawled.
An easy way to get links that they may not know about otherwise, and as it says in the privacy st. if you do not want to be tracked turn it off.
Links dont always mean popular whereas if a site is getting a million visits a day that means popular. If google notice that a high percentage of visits occur to foo.com then foo.com is popular and deserves to be ranked high in serps
Perhaps that are using a combination of PR and (real) popularity to rank sites.
I checked the log files recently and what do you know? Googlebot came sniffing around soon after I visited the page.
I have been hearing from others that this has really kicked in these past couple weeks.
I did check the site out with IE on a machine with the toolbar, which I believe is the only way it could have been found that quickly.
Forgive me if this question has been asked... What happens if you simply hide the toolbar? Searches run on Google with the toolbar hidden do not show up in the toolbar search history, so it seems reasonable to assume the toolbar is inactivated when it doesn't display.
To sum up from my point of view:
- There are many many ways a url can become known and then crawled.
- To the best of my knowledge, the toolbar is not currently one of those ways a url becomes known.
- To the best of my knowledge, anyone who is certain that the toolbar has caused their page to be crawled is therefore mistaken.
- If you turn off the advanced features, the toolbar is completely inert and does not report any info to Google.
And these two I would add on an unofficial basis:
- It's my personal, unofficial belief that using toolbar data in the future to augment our crawl is not only a good idea, but specifically allowed by the original policies we posted.
- To the best of my knowledge, no one has ever been forced to install the toolbar. If the toolbar worries you, then just don't install it.
