Welcome to WebmasterWorld Guest from 54.198.46.95

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

abot/0.1

New name for npbot.

     
6:31 pm on Mar 31, 2005 (gmt 0)

10+ Year Member



Old ua:
np/0.1_(np;_http://www.nameprotect.com;_npbot@nameprotect.com)

New name:
abot/0.1_(abot;_http://www.abot.com;_abot@abot.com)

IP: 24.177.134.6

7:05 pm on Mar 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Very nice catch, fiestagirl!
7:33 pm on Mar 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yep - the gobsh1tes in NP decided to hit my site fairly hard this morning. (It is the main reference site on Irish (.ie) domain names.) I had to add a permanent IP ban. I was wondering who abot.com was.A check on the abot.com hoster from their domain name shows 4851 domains registered as of March 01. The nameprotect.com hoster shows 2010 domains hosted. An IP level ban is so much more efficient than having to ban each potential spoof incarnation individually.

Regards...jmcc

6:13 pm on Apr 1, 2005 (gmt 0)

10+ Year Member



hummm....

Deny from 24.177.134.6

doesn't stop them.

7:09 pm on Apr 1, 2005 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



try deny from 24.177.
or
RewriteCond %{REMOTE_ADDR} ^24\.177\.1(2[89]¦3[0-5])\. [OR]

the deny you used with the last Class of (6) is both too narrow and specific.

11:04 pm on Apr 1, 2005 (gmt 0)

10+ Year Member



or:
Deny from 24.177.128.0/21
1:04 pm on Apr 2, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



24.177.134.6 - - [01/Apr/2005:22:15:42 -0800] "GET /robots.txt HTTP/1.0" 200 2200 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"
24.177.134.6 - - [01/Apr/2005:22:15:43 -0800] "GET / HTTP/1.0" 403 480 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"
24.177.134.6 - - [01/Apr/2005:22:26:08 -0800] "GET /Blah.html HTTP/1.0" 403 480 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"

A rose by any other name...

5:28 am on Apr 16, 2005 (gmt 0)

10+ Year Member



Hi all,

The CIDR ranges and single IP below cover everything I've found for Nameprotect / NP / NPbot / aibot / abot. If anyone has any more let me know.

24.177.134.6
12.175.0.32/28
12.148.209.192/26
12.148.196.128/25
69.8.156.192/26

5:56 pm on Apr 27, 2005 (gmt 0)

10+ Year Member



I have blocked these guys through my firewall, and through my .htaccess however somehow they keep coming through. I just dont understand how they can do this?

Any thoughts?

Cheers.

1:03 pm on May 3, 2005 (gmt 0)

10+ Year Member



Can someone explain what aipbot/ nameprotect is? hmmm rephrase - why it needs blocking? It seems to go through my site a few times a week and I was just trying to research it when I found this thread
1:07 pm on May 3, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Their site is pretty clear:

[nameprotect.com...]

Their bot scans your site for copyright infractions related to their clients.

1:10 pm on May 3, 2005 (gmt 0)

10+ Year Member



ok, so basically like a spy/ detective? Thanks
7:05 pm on May 4, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It's been my experience that they try each UA on your site, if they are blocked they switch to a generic Windows + IE UA, so you have to resort to blocking their ranges.
2:13 am on May 5, 2005 (gmt 0)

10+ Year Member



aipbot seems to have started to take a "if I behave I'll be let back in" approach in the last week or so. Previously it would read robots.txt and ignore it, persevering for a few 403'd pages. But now it just reads robots.txt, in which it is excluded.
8:02 pm on May 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Has anyone seen this?

24.177.134.6 - - [07/May/2005:03:08:02 -0400] "GET /robots.txt HTTP/1.0" 403 4429 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"

202.96.140.119 - - [07/May/2005:04:14:44 -0400] "GET / HTTP/1.0" 403 4429 "-" "AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , China)"

I'm confused. Ban both blocks? Are both scrapers, or what ever you want to call them, et. al.

1:18 pm on May 16, 2005 (gmt 0)

10+ Year Member



The top one (24.177.134.6) has been battering my site on occasion for over a week. I set up a deny from today and the server load went from an average of 3.5 back down to 0.6.

Whether it's legitimate or not it was causing slowness on my server.

The page linked in the UA doesn't give any useful information, aside from what to put in the robots.txt file. No way to contact them to ask them to hold back on the page grabbing.

4:36 pm on May 16, 2005 (gmt 0)

10+ Year Member



donovanh,

The IP you mention IS another manifestation of Nameprotect. It hit me first with the UA NPbot (Nameprotect's bot) then as aipbot a minute later and has stayed the same ever since.

Nameprotect seem to be widely disliked for excessive spidering, and I assume the use of another UA and IP is to get round those webmasters who've banned them on their 'official' IP/UA. They also use UA's from standard browsers from time to time on the IP's mentioned in in this thread, and I suppose they may well use others.

9:00 pm on May 17, 2005 (gmt 0)

10+ Year Member



I sent an email to the e-mail address in the UA asking them to slow it down, but the e-mail has been delayed and doesn't seem to have been delivered.

It tried to grab 17,000 pages in a couple of hours, not enough to push it over, but quite a noticable strain on the server.

I'll be keeping an eye out for other manifestations.

9:23 pm on May 25, 2005 (gmt 0)

10+ Year Member



These guys are not playing nice with me either.

I had

User-agent: nameprotect
Disallow: /

in my robots. then i went to NPBot. Now they are showing up in my logs as:

c24.177.134.6.static.ncr.charter.com

They just keep remorphing themselves so they can ignore the robots file. I'm gonna ban the whole ip range.

2:34 pm on May 26, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> They just keep remorphing themselves so they can ignore the robots file.

Robots.txt compliance is optional in any case. They morph so you'll have difficulty banning their user agent via Mod_Rewrite or a straight deny from .htaccess ban.

> I'm gonna ban the whole ip range.

The only effective solution for now... until they lease new IP addresses.

3:16 am on May 29, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



These jerks have sent bogus DMCA notices to many Amazon associates. I strongly urge everyone to not only block their access, but when they go around the blocks send them a cease and desist letter.
4:31 am on May 29, 2005 (gmt 0)

10+ Year Member



These jerks have sent bogus DMCA notices to many Amazon associates.

Whats in it for them? Touting for business?

5:03 am on May 29, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I know what Classless Inter-Domain Routing (CIDR) is, but despite reading several tutorials I still don't understand how to translate them to a network range.

For example, I want to block the 69.8.156.192/26 CIDR for NameProtect. Can this be translated to a traditional IP Address/Subnet Mask?

Thanks. :)

5:18 am on May 29, 2005 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Typed this up the other day. It's a reverse lookup on what you need, but maybe it'll help.

Subtract the start address of the range from the end address, keeping each dot-separated-column separate.

Look up the Netmask or CIDR based on this Difference.


Difference Addresses CIDR Netmask
0. 0. 0. 0 1 32 255.255.255.255
0. 0. 0. 1 2 31 255.255.255.254
0. 0. 0. 3 4 30 255.255.255.252
0. 0. 0. 7 8 29 255.255.255.248
0. 0. 0. 15 16 28 255.255.255.240
0. 0. 0. 31 32 27 255.255.255.224
0. 0. 0. 63 64 26 255.255.255.192
0. 0. 0.127 128 25 255.255.255.128
0. 0. 0.255 256 24 255.255.255.0

0. 0. 1.255 512 23 255.255.254.0
0. 0. 3.255 1,024 22 255.255.252.0
0. 0. 7.255 2,048 21 255.255.248.0
0. 0. 15.255 4,096 20 255.255.240.0
0. 0. 31.255 8,192 19 255.255.224.0
0. 0. 63.255 16,383 18 255.255.192.0
0. 0.127.255 32,767 17 255.255.128.0
0. 0.255.255 65,535 16 255.255.0.0

0. 1.255.255 131,072 15 255.254.0.0
0. 3.255.255 262,144 14 255.252.0.0
0. 7.255.255 524,288 13 255.248.0.0
0. 15.255.255 1,048,576 12 255.240.0.0
0. 31.255.255 2,097,152 11 255.224.0.0
0. 63.255.255 4,194,304 10 255.192.0.0
0.127.255.255 8,388,608 9 255.128.0.0
0.255.255.255 16,777,216 8 255.0.0.0

1.255.255.255 33,554,432 7 254.0.0.0
3.255.255.255 67,108,864 6 242.0.0.0
7.255.255.255 134,217,728 5 248.0.0.0
15.255.255.255 268,435,456 4 240.0.0.0
31.255.255.255 536,870,912 3 224.0.0.0
63.255.255.255 1,073,741,824 2 192.0.0.0
127.255.255.255 2,147,483,648 1 128.0.0.0
255.255.255.255 4,294,967,296 0 0.0.0.0.0

Hopefully, the formatting won't be too messed-up by posting.

You might also try a search for CIDR Network Netmask converter... There are several out there.

Jim

6:22 am on May 29, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Jim.

The formatting was awful. Or maybe it was the Rum. :)

I found a freeware calculator. I don't think I understand it though.

Clearly I'm off-topic and I appreciate your indulgence but I'll pick a more appropriate forum to ask for more help.

12:54 pm on May 29, 2005 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



For example, I want to block the 69.8.156.192/26 CIDR for NameProtect. Can this be translated to a traditional IP Address/Subnet Mask?

This page has a free conversion tool.
Second column, 2nd last box near bottom.
[dnsstuff.com...]

provides the following:
[dnsstuff.com...]

3:55 am on May 30, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Great site. Thanks wilderness.
3:26 am on Jun 2, 2005 (gmt 0)

10+ Year Member



Sounds like you have the mask aready, but in case... a /26 cidr notation corresponds to a 255.255.255.192 subnet mask.

All commercial spiders utilize your commercial resources for their commercial benefit. With the major search engines it is mostly win-win because they generate trafic in return. With many other commercial bots there is no upside. I always ban commercial bots where there is no upside without particular regard to what their commercial interests are.

4:25 am on Jun 2, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thank you, idoc.

I know about user agents being as I publish the web's only regularly updated browscap.ini file.

What I'm not so good at is the technical side of things like translating a CIDR into something my software can make use of. :)

 

Featured Threads

Hot Threads This Week

Hot Threads This Month