Problems with Voila

Can't stop it, hundreds of thousands of hits per day.

[motorhaven]

The French search engine Voila is driving me crazy. Its Voilabot won't stop!

I've tried banning it in robots.txt and it ends up not fetching pages but instead fetches robots.txt 10,000 - 20,000 times per day.

I've tried firewalling it with both rejected and dropped connections: but its too stupid and just tries to reconnect several about 100,000 - 200,000 times per day.

I've tried giving it 404 error codes for the pages, but again its stupid and just retries the pages countless times.

I've tried redirecting the bot with a 301 and 302 back to itself.... no luck. It still floods us.

I've tried contacting them politely... no response.
I've tried contacting them not so politely... no response.
I've threatened legal action... no response.

Any suggestions? This has been going on for months!

[volatilegx]

I'd contact your upstream provider and ask them to ban Voila's IP range at their switch.

[motorhaven]

Already tried that.... they said they can't (won't) do it because they will not block a "legitimate" crawler from their other clients.

I'm going to try 403, 401 and 410 error codes.... plus contact their upstream and complain.

[volatilegx]

I think 403's are the way to go. It's easy to ban the IPs (and the user agent) from the entire site. Put this in a .htaccess file:

SetEnvIf User-Agent VoilaBot ban_me
<files *>
Order Deny,Allow
Allow from all
Deny from env=ban_me
Deny from 193.252.148.208
Deny from 193.252.148.209
Deny from 195.101.94.80
Deny from 195.101.94.81
Deny from 195.101.94.15
Deny from 195.101.94.101
Deny from 195.101.94.166
Deny from 195.101.94.208
Deny from 195.101.94.209
Deny from 209.185.188.207
Deny from 216.35.76.11
</files>

The IPs I listed are the ones I have recorded for VoilaBot