Welcome to WebmasterWorld Guest from

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

Does Mediapartners-Google feed Googlebot proper?



3:01 am on Jan 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Has anyone seen any evidence that Mediapartners-Google feeds it's close-friend Googlebot; thus saving Google huge amounts of bandwidth if Googlebot was interested in crawling a page anyway?

Whilst the two bots are for a different primary purpose, I don't see any reason why they shouldn't share page source code once retrieved (they can process it entirely differently of course).


3:42 am on Jan 16, 2004 (gmt 0)

10+ Year Member

Google tech have told me that the two bots are entirely separate. (This is good, because the AdSense bot sees more of my site than Google should).

While it would save Google some bandwidth, the two robots do very different things.


3:53 am on Jan 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Interesting, thanks James.

Seems a bit of a waste of effort on Google's part; there is absoltely no reason why a page retrieved by Mediapartners-Google should not feed content to the index based on Googlebot's robots.txt restrictions. Bizarre!

While it would save Google some bandwidth

It would also save _me_ some bandwidth; which was more to the point!


4:17 am on Jan 16, 2004 (gmt 0)

WebmasterWorld Senior Member jenstar is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Others have reported that sites are indexed and show up in the serps *after* mediabot but *before* googlebot ever visits the page. But the FAQ does not say anything about the two bots not sharing info, even though many would assume that from the way the FAQ section is actually worded.

Participating in Google AdSense does not affect your site's rank in Google search results. Google AdSense will not affect the search results we deliver. Google believes strongly in freedom of expression and therefore offers broad access to content across the web. Our search results are unbiased by our relationships with paying advertisers and publishers. We will continue to show search results according to our PageRank technology.

So it could be that mediabot does get specific pages spidered and into the serps, but that it doesn't make a difference of which bot it is that gathers the page information for the serps.


7:39 am on Jan 16, 2004 (gmt 0)

10+ Year Member

Section 16 of the terms also states:

In addition, You grant Google the right to access, index and cache the Site(s), or any portion thereof, including by automated means including Web spiders or crawlers.


10:35 am on Jan 16, 2004 (gmt 0)

10+ Year Member

The opposite is definitely true, so they are not completely separate.

I'm 100% sure mediabot uses data from googlebot, maybe from the cache.

I have a site with huge parts excluded via robots.txt, but there are adsense ads on the excluded pages, and I have never, not even once, seen mediabot touch the non-excluded pages. Mediabot only takes the excluded pages.

From a bandwidth point of view it makes sense to let mediabot use the data from the normal googlebot runs.


11:02 am on Jan 16, 2004 (gmt 0)

10+ Year Member

I'm a member of a private forum that is not in any of the search engine indexes. This was done intentionally. The owners decided to join AdSense a few months ago to help fund the operational cost. We receive targeted ads and not one page has ever shown up in the SERPS. I think as long as you set the robots.txt properly both bots abide by it. At least they have for us. I bet there are other similar sites whose visitors come in from other than SEs and do not want just anyone from the general public walking in but do want to generate revenues from the visitor base they developed.

Featured Threads

Hot Threads This Week

Hot Threads This Month