Forum Moderators: open

Message Too Old, No Replies

Blocking Cuil bots

         

Asia_Expat

10:27 pm on May 1, 2010 (gmt 0)

10+ Year Member



I've noticed some very aggressive spidering from Cuil IP ranges, so I want to block this stupid bot. Their IP ranges are as detailed here (allegedly)...
[cuil.com...]

Could someone please confirm the correct CIDR notation for those ranges for my firewall.

tangor

11:12 pm on May 1, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Got to ask... have you denied them in robots.txt? I get a bunch of inquiries, but cuil so far, has ONLY taken robots.txt... and that's a drop in the bucket...

Actually, I whitelist only a few bots, deny all others, and cuil honors it.

Asia_Expat

12:02 am on May 2, 2010 (gmt 0)

10+ Year Member



No, because I figured they would continue hammering my forum until the next time they read the robots.txt file (they seem to have got stuck in a particular forum thread which they've been loading over and over for the last day or two)... so I'll just scupper them in my firewall and be done with it.

tangor

12:08 am on May 2, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



That works, too!

tpeacock

8:31 pm on May 2, 2010 (gmt 0)

10+ Year Member



This is what I have for CIDR notation for Cuil IP ranges:

216.129.119.0/27
38.99.13.112/28
67.218.116.128/29
67.218.116.160/29
216.129.119.32/27
216.129.119.81

Thomas

Asia_Expat

10:40 pm on May 2, 2010 (gmt 0)

10+ Year Member



Thanks... that should make my firewall deny file look a bit smaller.

dstiles

2:13 am on May 4, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



They list their bot IPs on their web site (I wish all bots would!). So far I haven't seen any other IP with twiceler.

I haven't seen any aggressive hits. They're far fewer than, eg, yahoo and msn on my sites.

tangor

4:40 am on May 4, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Checked the ranges above... I've been hit by all... and the only file taken was robots.txt (which disallows cuil). 1,561 requests have come from all, sometimes two to five times a day, since the first of the year... Seems well behaved, at least in my experience.