We have a DHTML menu that google does not spider at all. We had to create a sitemap to get spidered. The problem is we had to modify for Google. If a search engine could provide good results from all sites they would beat out Google. There should be no such thing as SEO if Google was doing their job. One of the biggest online computer etailers does not have anything in Google but weird URLS that mean nothing (no title no description) for all their products. The big guys that use TV and other advertising just don't care about Google. Big companies just buy their ads. I am sure some people are glad this happens but it means that Google is missing a lot of good stuff. Search engines are or soon will be nothing but advertisments. The web is not what it used to be it is a huge consumer juggernaught.
|and maintains his position. |
Falling over in my seat at what i just read. I have never seen a more meaningless debate have such significance on WW.
why does everyone have to call it a conspiracy theory? "conspiring" to do things that cause business to grow is also called 'good business practices' by others.
I used to say it before, but now I don't - the phrase is over-abused.
|Well, there is a 1-to-1 correlation some how between visiting unindexed new pages with the toolbar and GoogleBot showing up. It has happened to too many people to be an accident. Some how they are linked - either directly from gbot putting the url in the spiders inbox, or via referral leaking - but I know for a fact, they are linked. |
I didn't believe the whole toolbar indexing thing...GoogleGuy's reassurances to the contrary were convincing, until now. I've had a site in development this past week...no incoming links. 4 pages of a subdomain (definitely no incoming links) were crawled, though none are indexed yet. I don't buy the referral log thing, since these pages didn't have any outgoing links at the time, either.
I don't see why people make such a big thing of it. It seems an obvious thing for Google to do.
Google aim to index as much of the web as possible; and would therefore want to use any source of URLs available to them.
If Toolbar reports a URL not previously known to Google they would be mad not to crawl it.
|I don't see why people make such a big thing of it. It seems an obvious thing for Google to do. |
Why would they deny it then? I agree, its not that big of an issue, I'd just like to know what the reality is, and my experience is leading me to question Google's version of reality.
There are some times when I don't want Google looking at what I'm looking at, though...site development being a good example.
|Why would they deny it then? |
That's what I don't understand!
They would be even more crazy not to index pages that the MediaPartners Googlebot comes after because those pages are definitely being read and seen by humans. Why would it not use that googlebot data once it's collected? I know they say it isn't, and I can't prove they do, but I can't imagine the two bots wouldn't share that data.
Okay, here's my experience.
A few weeks ago, I created a new forum (message board) on my site.
I do have the Google Toolbar (2.0 beta) on my system, and I also have AdSense ads on every page of my forum.
Here is what I observed repeatedly:
Within two minutes of creating a new discussion thread, a GoogleBot showed up to that exact (and previously non-existent) URL.
This, then, would normally indicate one of the following:
1) The URL was 'leaked' from the toolbar.
2) AdSense was 'inviting' the bot there.
3) Both of the above.
Since GG has unequivocally denied that #1 is the case, it must mean that he's either mistaken (not likely, IMHO) or lying (even less likely, IMHO) or that the answer is #2 (the bot visit is AdSense related).
Unfortunately, I'm currently too lazy to uninstall the toolbar from my browser to test this, but for those who are concerned about the mysterious appearance of Googlebots... perhaps you're also running AdSense ads on those pages?
Just something to think about.
Visiting a page with AdSense code on it may cause the Mediapartners bot to visit, but not the normal index bots, IMO.
GG >> Referral leaking is a common way that we find leaks to unindexed pages
So if I have the toolbar active my browser, go to my new website, then use the search window on the toolbar while I'm there, does that create a referral from the new site?
Holly GoogleBot Batman!
(since my other attempts of making a new thread were
refused, grumble, censorship, grumble, I will post here.)
65k pages downloaded since 1AM today, an average of 104 pages per minute, 1.7 pages per second. This onslaught is slowing my server from a normally zippy rabbit, to a slow laboring turtle.
My only hope is that this might mean an update to my 3 month old PR and backlink values.
Anyone else seeing this kind of traffic?
| This 42 message thread spans 2 pages: < < 42 ( 1  ) |