Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Weird bot behavior



3:49 pm on Jul 16, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Article [securitynewsportal.com]

Requests for:
[error] [client] File does not exist: /public_html/maillist/maillist_signin.asp

If this is Google, have they turned their technology loose on the web to create SPAM mailing list to sell in a quest for even greater profits?

paranoid or what, but this is what is out there.


4:35 pm on Jul 17, 2003 (gmt 0)

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member

We have a DHTML menu that google does not spider at all. We had to create a sitemap to get spidered. The problem is we had to modify for Google. If a search engine could provide good results from all sites they would beat out Google. There should be no such thing as SEO if Google was doing their job. One of the biggest online computer etailers does not have anything in Google but weird URLS that mean nothing (no title no description) for all their products. The big guys that use TV and other advertising just don't care about Google. Big companies just buy their ads. I am sure some people are glad this happens but it means that Google is missing a lot of good stuff. Search engines are or soon will be nothing but advertisments. The web is not what it used to be it is a huge consumer juggernaught.


12:59 am on Jul 18, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

and maintains his position.

i know for a fact

Falling over in my seat at what i just read. I have never seen a more meaningless debate have such significance on WW.


6:40 am on Jul 19, 2003 (gmt 0)

10+ Year Member

why does everyone have to call it a conspiracy theory? "conspiring" to do things that cause business to grow is also called 'good business practices' by others.

I used to say it before, but now I don't - the phrase is over-abused.


3:41 pm on Jul 20, 2003 (gmt 0)

10+ Year Member

Well, there is a 1-to-1 correlation some how between visiting unindexed new pages with the toolbar and GoogleBot showing up. It has happened to too many people to be an accident. Some how they are linked - either directly from gbot putting the url in the spiders inbox, or via referral leaking - but I know for a fact, they are linked.

I didn't believe the whole toolbar indexing thing...GoogleGuy's reassurances to the contrary were convincing, until now. I've had a site in development this past week...no incoming links. 4 pages of a subdomain (definitely no incoming links) were crawled, though none are indexed yet. I don't buy the referral log thing, since these pages didn't have any outgoing links at the time, either.


3:45 pm on Jul 20, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I don't see why people make such a big thing of it. It seems an obvious thing for Google to do.

Google aim to index as much of the web as possible; and would therefore want to use any source of URLs available to them.

If Toolbar reports a URL not previously known to Google they would be mad not to crawl it.


3:57 pm on Jul 20, 2003 (gmt 0)

10+ Year Member

I don't see why people make such a big thing of it. It seems an obvious thing for Google to do.

Why would they deny it then? I agree, its not that big of an issue, I'd just like to know what the reality is, and my experience is leading me to question Google's version of reality.

There are some times when I don't want Google looking at what I'm looking at, though...site development being a good example.


4:04 pm on Jul 20, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Why would they deny it then?

That's what I don't understand!

Although to be fair I don't think Google have denied it as such, more GG stated that he didn't _think_ it happened; and also went on to state that nothing in their privacy policy would prevent them from doing so.

If you ask me it's not the sort of thing that has anything to do with a Privacy Policy anyway. A URL is a URL; if the web service handling Toolbar queries simply chucks each URL into Googlebot's Inbox with no reference to the Toolbar user that introduced it then there is no reason for it to even come up within a privacy statement.


11:57 pm on Jul 20, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

They would be even more crazy not to index pages that the MediaPartners Googlebot comes after because those pages are definitely being read and seen by humans. Why would it not use that googlebot data once it's collected? I know they say it isn't, and I can't prove they do, but I can't imagine the two bots wouldn't share that data.


6:46 pm on Jul 29, 2003 (gmt 0)

10+ Year Member

Okay, here's my experience.

A few weeks ago, I created a new forum (message board) on my site.

I do have the Google Toolbar (2.0 beta) on my system, and I also have AdSense ads on every page of my forum.

Here is what I observed repeatedly:
Within two minutes of creating a new discussion thread, a GoogleBot showed up to that exact (and previously non-existent) URL.

This, then, would normally indicate one of the following:
1) The URL was 'leaked' from the toolbar.
2) AdSense was 'inviting' the bot there.
3) Both of the above.

Since GG has unequivocally denied that #1 is the case, it must mean that he's either mistaken (not likely, IMHO) or lying (even less likely, IMHO) or that the answer is #2 (the bot visit is AdSense related).

Unfortunately, I'm currently too lazy to uninstall the toolbar from my browser to test this, but for those who are concerned about the mysterious appearance of Googlebots... perhaps you're also running AdSense ads on those pages?

Just something to think about.


6:56 pm on Jul 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Visiting a page with AdSense code on it may cause the Mediapartners bot to visit, but not the normal index bots, IMO.


7:32 pm on Jul 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

GG >> Referral leaking is a common way that we find leaks to unindexed pages

So if I have the toolbar active my browser, go to my new website, then use the search window on the toolbar while I'm there, does that create a referral from the new site?


8:03 pm on Jul 29, 2003 (gmt 0)

10+ Year Member

Holly GoogleBot Batman!

(since my other attempts of making a new thread were
refused, grumble, censorship, grumble, I will post here.)

65k pages downloaded since 1AM today, an average of 104 pages per minute, 1.7 pages per second. This onslaught is slowing my server from a normally zippy rabbit, to a slow laboring turtle.

My only hope is that this might mean an update to my 3 month old PR and backlink values.

Anyone else seeing this kind of traffic?

This 42 message thread spans 2 pages: 42

Featured Threads

Hot Threads This Week

Hot Threads This Month