homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google AdSense
Forum Library, Charter, Moderators: incrediBILL & martinibuster

Google AdSense Forum

Does Mediapartners-Google feed Googlebot proper?

WebmasterWorld Senior Member 10+ Year Member

Msg#: 1446 posted 3:01 am on Jan 16, 2004 (gmt 0)

Has anyone seen any evidence that Mediapartners-Google feeds it's close-friend Googlebot; thus saving Google huge amounts of bandwidth if Googlebot was interested in crawling a page anyway?

Whilst the two bots are for a different primary purpose, I don't see any reason why they shouldn't share page source code once retrieved (they can process it entirely differently of course).



10+ Year Member

Msg#: 1446 posted 3:42 am on Jan 16, 2004 (gmt 0)

Google tech have told me that the two bots are entirely separate. (This is good, because the AdSense bot sees more of my site than Google should).

While it would save Google some bandwidth, the two robots do very different things.


WebmasterWorld Senior Member 10+ Year Member

Msg#: 1446 posted 3:53 am on Jan 16, 2004 (gmt 0)

Interesting, thanks James.

Seems a bit of a waste of effort on Google's part; there is absoltely no reason why a page retrieved by Mediapartners-Google should not feed content to the index based on Googlebot's robots.txt restrictions. Bizarre!

While it would save Google some bandwidth

It would also save _me_ some bandwidth; which was more to the point!


WebmasterWorld Senior Member jenstar us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 1446 posted 4:17 am on Jan 16, 2004 (gmt 0)

Others have reported that sites are indexed and show up in the serps *after* mediabot but *before* googlebot ever visits the page. But the FAQ does not say anything about the two bots not sharing info, even though many would assume that from the way the FAQ section is actually worded.

Participating in Google AdSense does not affect your site's rank in Google search results. Google AdSense will not affect the search results we deliver. Google believes strongly in freedom of expression and therefore offers broad access to content across the web. Our search results are unbiased by our relationships with paying advertisers and publishers. We will continue to show search results according to our PageRank technology.

So it could be that mediabot does get specific pages spidered and into the serps, but that it doesn't make a difference of which bot it is that gathers the page information for the serps.


10+ Year Member

Msg#: 1446 posted 7:39 am on Jan 16, 2004 (gmt 0)

Section 16 of the terms also states:

In addition, You grant Google the right to access, index and cache the Site(s), or any portion thereof, including by automated means including Web spiders or crawlers.


10+ Year Member

Msg#: 1446 posted 10:35 am on Jan 16, 2004 (gmt 0)

The opposite is definitely true, so they are not completely separate.

I'm 100% sure mediabot uses data from googlebot, maybe from the cache.

I have a site with huge parts excluded via robots.txt, but there are adsense ads on the excluded pages, and I have never, not even once, seen mediabot touch the non-excluded pages. Mediabot only takes the excluded pages.

From a bandwidth point of view it makes sense to let mediabot use the data from the normal googlebot runs.


10+ Year Member

Msg#: 1446 posted 11:02 am on Jan 16, 2004 (gmt 0)

I'm a member of a private forum that is not in any of the search engine indexes. This was done intentionally. The owners decided to join AdSense a few months ago to help fund the operational cost. We receive targeted ads and not one page has ever shown up in the SERPS. I think as long as you set the robots.txt properly both bots abide by it. At least they have for us. I bet there are other similar sites whose visitors come in from other than SEs and do not want just anyone from the general public walking in but do want to generate revenues from the visitor base they developed.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google AdSense
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved