Welcome to WebmasterWorld Guest from 54.147.50.227

Forum Moderators: open

Message Too Old, No Replies

Weird bot behavior

     
3:49 pm on Jul 16, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 6, 2001
posts:2213
votes: 0


Article [securitynewsportal.com]

Requests for:
[error] [client 64.68.82.31] File does not exist: /public_html/maillist/maillist_signin.asp

If this is Google, have they turned their technology loose on the web to create SPAM mailing list to sell in a quest for even greater profits?

paranoid or what, but this is what is out there.

4:35 pm on July 17, 2003 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 14, 2003
posts:4281
votes: 25


We have a DHTML menu that google does not spider at all. We had to create a sitemap to get spidered. The problem is we had to modify for Google. If a search engine could provide good results from all sites they would beat out Google. There should be no such thing as SEO if Google was doing their job. One of the biggest online computer etailers does not have anything in Google but weird URLS that mean nothing (no title no description) for all their products. The big guys that use TV and other advertising just don't care about Google. Big companies just buy their ads. I am sure some people are glad this happens but it means that Google is missing a lot of good stuff. Search engines are or soon will be nothing but advertisments. The web is not what it used to be it is a huge consumer juggernaught.
12:59 am on July 18, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 2002
posts:1056
votes: 0


and maintains his position.

i know for a fact

Falling over in my seat at what i just read. I have never seen a more meaningless debate have such significance on WW.

6:40 am on July 19, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 5, 2002
posts:529
votes: 0


why does everyone have to call it a conspiracy theory? "conspiring" to do things that cause business to grow is also called 'good business practices' by others.

I used to say it before, but now I don't - the phrase is over-abused.

3:41 pm on July 20, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 20, 2003
posts:390
votes: 0


Well, there is a 1-to-1 correlation some how between visiting unindexed new pages with the toolbar and GoogleBot showing up. It has happened to too many people to be an accident. Some how they are linked - either directly from gbot putting the url in the spiders inbox, or via referral leaking - but I know for a fact, they are linked.

I didn't believe the whole toolbar indexing thing...GoogleGuy's reassurances to the contrary were convincing, until now. I've had a site in development this past week...no incoming links. 4 pages of a subdomain (definitely no incoming links) were crawled, though none are indexed yet. I don't buy the referral log thing, since these pages didn't have any outgoing links at the time, either.

3:45 pm on July 20, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2003
posts:1633
votes: 0


I don't see why people make such a big thing of it. It seems an obvious thing for Google to do.

Google aim to index as much of the web as possible; and would therefore want to use any source of URLs available to them.

If Toolbar reports a URL not previously known to Google they would be mad not to crawl it.

3:57 pm on July 20, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 20, 2003
posts:390
votes: 0


I don't see why people make such a big thing of it. It seems an obvious thing for Google to do.

Why would they deny it then? I agree, its not that big of an issue, I'd just like to know what the reality is, and my experience is leading me to question Google's version of reality.

There are some times when I don't want Google looking at what I'm looking at, though...site development being a good example.

4:04 pm on July 20, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2003
posts:1633
votes: 0


Why would they deny it then?

That's what I don't understand!

Although to be fair I don't think Google have denied it as such, more GG stated that he didn't _think_ it happened; and also went on to state that nothing in their privacy policy would prevent them from doing so.

If you ask me it's not the sort of thing that has anything to do with a Privacy Policy anyway. A URL is a URL; if the web service handling Toolbar queries simply chucks each URL into Googlebot's Inbox with no reference to the Toolbar user that introduced it then there is no reason for it to even come up within a privacy statement.

11:57 pm on July 20, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 8, 2002
posts:2335
votes: 0


They would be even more crazy not to index pages that the MediaPartners Googlebot comes after because those pages are definitely being read and seen by humans. Why would it not use that googlebot data once it's collected? I know they say it isn't, and I can't prove they do, but I can't imagine the two bots wouldn't share that data.
6:46 pm on July 29, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 23, 2002
posts:532
votes: 0


Okay, here's my experience.

A few weeks ago, I created a new forum (message board) on my site.

I do have the Google Toolbar (2.0 beta) on my system, and I also have AdSense ads on every page of my forum.

Here is what I observed repeatedly:
Within two minutes of creating a new discussion thread, a GoogleBot showed up to that exact (and previously non-existent) URL.

This, then, would normally indicate one of the following:
1) The URL was 'leaked' from the toolbar.
2) AdSense was 'inviting' the bot there.
3) Both of the above.

Since GG has unequivocally denied that #1 is the case, it must mean that he's either mistaken (not likely, IMHO) or lying (even less likely, IMHO) or that the answer is #2 (the bot visit is AdSense related).

Unfortunately, I'm currently too lazy to uninstall the toolbar from my browser to test this, but for those who are concerned about the mysterious appearance of Googlebots... perhaps you're also running AdSense ads on those pages?

Just something to think about.

6:56 pm on July 29, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2002
posts:741
votes: 0


Visiting a page with AdSense code on it may cause the Mediapartners bot to visit, but not the normal index bots, IMO.
7:32 pm on July 29, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2000
posts:1133
votes: 0


GG >> Referral leaking is a common way that we find leaks to unindexed pages

So if I have the toolbar active my browser, go to my new website, then use the search window on the toolbar while I'm there, does that create a referral from the new site?

8:03 pm on July 29, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 10, 2003
posts:157
votes: 0


Holly GoogleBot Batman!

(since my other attempts of making a new thread were
refused, grumble, censorship, grumble, I will post here.)

65k pages downloaded since 1AM today, an average of 104 pages per minute, 1.7 pages per second. This onslaught is slowing my server from a normally zippy rabbit, to a slow laboring turtle.

My only hope is that this might mean an update to my 3 month old PR and backlink values.

Anyone else seeing this kind of traffic?

This 42 message thread spans 2 pages: 42