homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 42 message thread spans 2 pages: < < 42 ( 1 [2]     
Weird bot behavior

 3:49 pm on Jul 16, 2003 (gmt 0)

Article [securitynewsportal.com]

Requests for:
[error] [client] File does not exist: /public_html/maillist/maillist_signin.asp

If this is Google, have they turned their technology loose on the web to create SPAM mailing list to sell in a quest for even greater profits?

paranoid or what, but this is what is out there.



 4:35 pm on Jul 17, 2003 (gmt 0)

We have a DHTML menu that google does not spider at all. We had to create a sitemap to get spidered. The problem is we had to modify for Google. If a search engine could provide good results from all sites they would beat out Google. There should be no such thing as SEO if Google was doing their job. One of the biggest online computer etailers does not have anything in Google but weird URLS that mean nothing (no title no description) for all their products. The big guys that use TV and other advertising just don't care about Google. Big companies just buy their ads. I am sure some people are glad this happens but it means that Google is missing a lot of good stuff. Search engines are or soon will be nothing but advertisments. The web is not what it used to be it is a huge consumer juggernaught.


 12:59 am on Jul 18, 2003 (gmt 0)

and maintains his position.

i know for a fact

Falling over in my seat at what i just read. I have never seen a more meaningless debate have such significance on WW.


 6:40 am on Jul 19, 2003 (gmt 0)

why does everyone have to call it a conspiracy theory? "conspiring" to do things that cause business to grow is also called 'good business practices' by others.

I used to say it before, but now I don't - the phrase is over-abused.


 3:41 pm on Jul 20, 2003 (gmt 0)

Well, there is a 1-to-1 correlation some how between visiting unindexed new pages with the toolbar and GoogleBot showing up. It has happened to too many people to be an accident. Some how they are linked - either directly from gbot putting the url in the spiders inbox, or via referral leaking - but I know for a fact, they are linked.

I didn't believe the whole toolbar indexing thing...GoogleGuy's reassurances to the contrary were convincing, until now. I've had a site in development this past week...no incoming links. 4 pages of a subdomain (definitely no incoming links) were crawled, though none are indexed yet. I don't buy the referral log thing, since these pages didn't have any outgoing links at the time, either.


 3:45 pm on Jul 20, 2003 (gmt 0)

I don't see why people make such a big thing of it. It seems an obvious thing for Google to do.

Google aim to index as much of the web as possible; and would therefore want to use any source of URLs available to them.

If Toolbar reports a URL not previously known to Google they would be mad not to crawl it.


 3:57 pm on Jul 20, 2003 (gmt 0)

I don't see why people make such a big thing of it. It seems an obvious thing for Google to do.

Why would they deny it then? I agree, its not that big of an issue, I'd just like to know what the reality is, and my experience is leading me to question Google's version of reality.

There are some times when I don't want Google looking at what I'm looking at, though...site development being a good example.


 4:04 pm on Jul 20, 2003 (gmt 0)

Why would they deny it then?

That's what I don't understand!

Although to be fair I don't think Google have denied it as such, more GG stated that he didn't _think_ it happened; and also went on to state that nothing in their privacy policy would prevent them from doing so.

If you ask me it's not the sort of thing that has anything to do with a Privacy Policy anyway. A URL is a URL; if the web service handling Toolbar queries simply chucks each URL into Googlebot's Inbox with no reference to the Toolbar user that introduced it then there is no reason for it to even come up within a privacy statement.


 11:57 pm on Jul 20, 2003 (gmt 0)

They would be even more crazy not to index pages that the MediaPartners Googlebot comes after because those pages are definitely being read and seen by humans. Why would it not use that googlebot data once it's collected? I know they say it isn't, and I can't prove they do, but I can't imagine the two bots wouldn't share that data.


 6:46 pm on Jul 29, 2003 (gmt 0)

Okay, here's my experience.

A few weeks ago, I created a new forum (message board) on my site.

I do have the Google Toolbar (2.0 beta) on my system, and I also have AdSense ads on every page of my forum.

Here is what I observed repeatedly:
Within two minutes of creating a new discussion thread, a GoogleBot showed up to that exact (and previously non-existent) URL.

This, then, would normally indicate one of the following:
1) The URL was 'leaked' from the toolbar.
2) AdSense was 'inviting' the bot there.
3) Both of the above.

Since GG has unequivocally denied that #1 is the case, it must mean that he's either mistaken (not likely, IMHO) or lying (even less likely, IMHO) or that the answer is #2 (the bot visit is AdSense related).

Unfortunately, I'm currently too lazy to uninstall the toolbar from my browser to test this, but for those who are concerned about the mysterious appearance of Googlebots... perhaps you're also running AdSense ads on those pages?

Just something to think about.


 6:56 pm on Jul 29, 2003 (gmt 0)

Visiting a page with AdSense code on it may cause the Mediapartners bot to visit, but not the normal index bots, IMO.


 7:32 pm on Jul 29, 2003 (gmt 0)

GG >> Referral leaking is a common way that we find leaks to unindexed pages

So if I have the toolbar active my browser, go to my new website, then use the search window on the toolbar while I'm there, does that create a referral from the new site?


 8:03 pm on Jul 29, 2003 (gmt 0)

Holly GoogleBot Batman!

(since my other attempts of making a new thread were
refused, grumble, censorship, grumble, I will post here.)

65k pages downloaded since 1AM today, an average of 104 pages per minute, 1.7 pages per second. This onslaught is slowing my server from a normally zippy rabbit, to a slow laboring turtle.

My only hope is that this might mean an update to my 3 month old PR and backlink values.

Anyone else seeing this kind of traffic?

This 42 message thread spans 2 pages: < < 42 ( 1 [2]
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved