Fundamental scaling issue w/ Adsense model

Can it possibly scale?

[Clark]

The call to the Mediapartners bot happens when someone views a page. There are many webmasters placing adsense code on dynamic pages. There are many search engines with spiders. There are many email harvest bots.

Question: Are you allowed to have adsense on a page blocked by robots.txt? (I think that's covered and the answer is no, correct?)

So essentially, google will have to spider ever more urls. An insane number. If a bot gets trapped in some loop on dynamic pages it can get crazy with useless URLs. How can G possibly keep up?

Then when you have a change to the page, it is important that mediabot refreshes the page for relevant content in a timely manner.

If it was just about a visit and they didn't need to keep any of the data, it would be easier. But the Mediabot probably has to be faster and more complete than the regular googlebot. And it needs to keep data on each url it visits so it can provide relevant content.

To me this will be one of the biggest challenges google faces in scaling the system.

[Jenstar]

You can't put it on a page that is blocked by robots.txt. The bot used by AdSense doesn't follow links, so it shouldn't get "trapped" so top speak. But if AdSense is placed on page(s) that are password protected, or not accessible for whatever reason, it will continue to try and fetch that URL to target it for AdSense.

People have seen lots of problems with session IDs - the same thread will be spidered repeatedly with each viewer's session ID. The bandwidth will eventually ad up, especially on super-active SID sites.

[Clark]

I'm not taking about Mediabot getting trapped by links. I'm saying that if another bot gets trapped into a loop, it will call the page which in turn will call the Mediabot. So Mediabot will still get called to visit bad pages. (though just now thinking about it, they probably get the IP of the bot and if it hits too many pages at once they can figure out it was a bot. Except for the harvest bots that change referer IPs each visit).

[linear]

The call to the Mediapartners bot happens when someone views a page.

Not every time someone calls the page. That would never scale.

Displaying the ad is completely a different process that doesn't involve the Mediapartners bot, so even if the rogue bot did go haywire, no Mediapartners issues would arise.

[freeflight2]

from a technical standpoint there is no scaling issue since you can spread the spidering (the 'bots') + serving of ads over thousands of servers - bandwidth, cpu and storage is getting cheaper and cheaper every day.

[figment88]

Hardly any bots execute javascript so the outlined scenario will not unfold.

[Clark]

ah, ok.