Just a word of caution to anyone using Overture. Watch out for the bot, crawler.searchmarketing.yahoo.com (IP 184.108.40.206).
This guy is currently behaving really badly, sending an exorbitant amount of simultaneous requests sustained over long periods of time. It brought our server down on its knees several times. Essentially it was acting as a Denial of Service attack.
We've had to temporarily ban the bot until Yahoo can tame this wild beast and make it be a bit more considerate [hey Yahoo, not everyone has a server farm you know].
well then it's a damn good spoofer because it only hits sites which are running overture ads, and it only hits the ad urls. And increased # of ads coincides with increased activity from this bot. I am told it is their automated ad checker.
It's the automated ad-checker ... but nobody at Y will acknowledge its existence. We have gone through a few extended battles with those bots over the past couple of years, and Y never admitted what was happening. The DoS aspect of the probing would certainly make for an interesting lawsuit ...
We started blocking the bots' IPs, and discovered that doing so kept us from receiving mail from SBC-Yahoo subscribers ... leading us to believe that not only are there unattended bots running from those servers, but that they are sharing the space with email users. When we unblock an IP based on an email message's headers so we can receive mail from that subscriber, the bots return.