Forum Moderators: open
The bot doesn't honor the crawl-delay in my robots.txt like msnbot. I've contacted MSN to see if there is any way to slow this sucker down but the people I've managed to get a hold of don't seem to know what I'm talking about... At this point, I've resorted to blocking the bot in my firewall but it's still causing issues for me.
The bot's visits coorelate directly to when I have my MSN AdCenter campaigns active and I don't want to turn them off. Does anyone know if this bot keeps coming or does it only come when you add new keywords? Has anyone had similar experiences with this bot?
[google.com...]
My problem is that I added a few hundred thousand keywords with distinct dynamic URLs rather quickly and it seems the MSNPTC bot wants to check all of my urls at a very fast rate.
When I pause my AdCenter Orders/Campaigns the bot stops and when I resume them, it comes back full force... I imagine it's still working through all of the keywords I've submitted and for now I'm staggering which campaigns I have on in order to limit the URLs the bot wants to crawl.
I'm wondering if this bot will continuously come back to recheck URLs that it's already checked or will it be satisfied with the initial crawl. i wouldn't be concerned if the site in question was on a heavier duty server but like I said, it's a side project for now and just can't handle the type of load this bot is driving.
I've had it denied for so long that I really don't pay any attention to what it attempts to read.
Sorry I can't help you more.
Perhaps somebody in one of the MSN forums or AD forums might be able to assit you better?
