| 3:42 am on Jan 16, 2004 (gmt 0)|
Google tech have told me that the two bots are entirely separate. (This is good, because the AdSense bot sees more of my site than Google should).
While it would save Google some bandwidth, the two robots do very different things.
| 3:53 am on Jan 16, 2004 (gmt 0)|
Interesting, thanks James.
Seems a bit of a waste of effort on Google's part; there is absoltely no reason why a page retrieved by Mediapartners-Google should not feed content to the index based on Googlebot's robots.txt restrictions. Bizarre!
|While it would save Google some bandwidth |
It would also save _me_ some bandwidth; which was more to the point!
| 4:17 am on Jan 16, 2004 (gmt 0)|
Others have reported that sites are indexed and show up in the serps *after* mediabot but *before* googlebot ever visits the page. But the FAQ does not say anything about the two bots not sharing info, even though many would assume that from the way the FAQ section is actually worded.
Participating in Google AdSense does not affect your site's rank in Google search results. Google AdSense will not affect the search results we deliver. Google believes strongly in freedom of expression and therefore offers broad access to content across the web. Our search results are unbiased by our relationships with paying advertisers and publishers. We will continue to show search results according to our PageRank technology.
So it could be that mediabot does get specific pages spidered and into the serps, but that it doesn't make a difference of which bot it is that gathers the page information for the serps.
| 7:39 am on Jan 16, 2004 (gmt 0)|
Section 16 of the terms also states:
|In addition, You grant Google the right to access, index and cache the Site(s), or any portion thereof, including by automated means including Web spiders or crawlers. |
| 10:35 am on Jan 16, 2004 (gmt 0)|
The opposite is definitely true, so they are not completely separate.
I'm 100% sure mediabot uses data from googlebot, maybe from the cache.
I have a site with huge parts excluded via robots.txt, but there are adsense ads on the excluded pages, and I have never, not even once, seen mediabot touch the non-excluded pages. Mediabot only takes the excluded pages.
From a bandwidth point of view it makes sense to let mediabot use the data from the normal googlebot runs.
| 11:02 am on Jan 16, 2004 (gmt 0)|
I'm a member of a private forum that is not in any of the search engine indexes. This was done intentionally. The owners decided to join AdSense a few months ago to help fund the operational cost. We receive targeted ads and not one page has ever shown up in the SERPS. I think as long as you set the robots.txt properly both bots abide by it. At least they have for us. I bet there are other similar sites whose visitors come in from other than SEs and do not want just anyone from the general public walking in but do want to generate revenues from the visitor base they developed.