Welcome to WebmasterWorld Guest from 54.196.232.162

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

Does Mediapartners-Google feed Googlebot proper?

     
3:01 am on Jan 16, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2003
posts:1633
votes: 0


Has anyone seen any evidence that Mediapartners-Google feeds it's close-friend Googlebot; thus saving Google huge amounts of bandwidth if Googlebot was interested in crawling a page anyway?

Whilst the two bots are for a different primary purpose, I don't see any reason why they shouldn't share page source code once retrieved (they can process it entirely differently of course).

3:42 am on Jan 16, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:June 28, 2003
posts:83
votes: 0


Google tech have told me that the two bots are entirely separate. (This is good, because the AdSense bot sees more of my site than Google should).

While it would save Google some bandwidth, the two robots do very different things.

3:53 am on Jan 16, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2003
posts:1633
votes: 0


Interesting, thanks James.

Seems a bit of a waste of effort on Google's part; there is absoltely no reason why a page retrieved by Mediapartners-Google should not feed content to the index based on Googlebot's robots.txt restrictions. Bizarre!

While it would save Google some bandwidth

It would also save _me_ some bandwidth; which was more to the point!

4:17 am on Jan 16, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jenstar is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 22, 2002
posts:4663
votes: 0


Others have reported that sites are indexed and show up in the serps *after* mediabot but *before* googlebot ever visits the page. But the FAQ does not say anything about the two bots not sharing info, even though many would assume that from the way the FAQ section is actually worded.

Participating in Google AdSense does not affect your site's rank in Google search results. Google AdSense will not affect the search results we deliver. Google believes strongly in freedom of expression and therefore offers broad access to content across the web. Our search results are unbiased by our relationships with paying advertisers and publishers. We will continue to show search results according to our PageRank technology.

So it could be that mediabot does get specific pages spidered and into the serps, but that it doesn't make a difference of which bot it is that gathers the page information for the serps.

7:39 am on Jan 16, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 2, 2002
posts:529
votes: 0


Section 16 of the terms also states:

In addition, You grant Google the right to access, index and cache the Site(s), or any portion thereof, including by automated means including Web spiders or crawlers.
10:35 am on Jan 16, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 28, 2002
posts:492
votes: 0


The opposite is definitely true, so they are not completely separate.

I'm 100% sure mediabot uses data from googlebot, maybe from the cache.

I have a site with huge parts excluded via robots.txt, but there are adsense ads on the excluded pages, and I have never, not even once, seen mediabot touch the non-excluded pages. Mediabot only takes the excluded pages.

From a bandwidth point of view it makes sense to let mediabot use the data from the normal googlebot runs.

11:02 am on Jan 16, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 11, 2003
posts:495
votes: 0


I'm a member of a private forum that is not in any of the search engine indexes. This was done intentionally. The owners decided to join AdSense a few months ago to help fund the operational cost. We receive targeted ads and not one page has ever shown up in the SERPS. I think as long as you set the robots.txt properly both bots abide by it. At least they have for us. I bet there are other similar sites whose visitors come in from other than SEs and do not want just anyone from the general public walking in but do want to generate revenues from the visitor base they developed.