Welcome to WebmasterWorld Guest from 54.158.55.251

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

abot/0.1

New name for npbot.

     
6:31 pm on Mar 31, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2002
posts:378
votes: 0


Old ua:
np/0.1_(np;_http://www.nameprotect.com;_npbot@nameprotect.com)

New name:
abot/0.1_(abot;_http://www.abot.com;_abot@abot.com)

IP: 24.177.134.6

7:05 pm on Mar 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


Very nice catch, fiestagirl!
7:33 pm on Mar 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 30, 2002
posts:2579
votes: 67


Yep - the gobsh1tes in NP decided to hit my site fairly hard this morning. (It is the main reference site on Irish (.ie) domain names.) I had to add a permanent IP ban. I was wondering who abot.com was.A check on the abot.com hoster from their domain name shows 4851 domains registered as of March 01. The nameprotect.com hoster shows 2010 domains hosted. An IP level ban is so much more efficient than having to ban each potential spoof incarnation individually.

Regards...jmcc

6:13 pm on Apr 1, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Dec 18, 2002
posts:257
votes: 0


hummm....

Deny from 24.177.134.6

doesn't stop them.

7:09 pm on Apr 1, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


try deny from 24.177.
or
RewriteCond %{REMOTE_ADDR} ^24\.177\.1(2[89]¦3[0-5])\. [OR]

the deny you used with the last Class of (6) is both too narrow and specific.

11:04 pm on Apr 1, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Dec 20, 2003
posts:268
votes: 0


or:
Deny from 24.177.128.0/21
1:04 pm on Apr 2, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
posts:1685
votes: 0


24.177.134.6 - - [01/Apr/2005:22:15:42 -0800] "GET /robots.txt HTTP/1.0" 200 2200 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"
24.177.134.6 - - [01/Apr/2005:22:15:43 -0800] "GET / HTTP/1.0" 403 480 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"
24.177.134.6 - - [01/Apr/2005:22:26:08 -0800] "GET /Blah.html HTTP/1.0" 403 480 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"

A rose by any other name...

5:28 am on Apr 16, 2005 (gmt 0)

New User

10+ Year Member

joined:Feb 29, 2004
posts:17
votes: 0


Hi all,

The CIDR ranges and single IP below cover everything I've found for Nameprotect / NP / NPbot / aibot / abot. If anyone has any more let me know.

24.177.134.6
12.175.0.32/28
12.148.209.192/26
12.148.196.128/25
69.8.156.192/26

5:56 pm on Apr 27, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 10, 2004
posts:126
votes: 0


I have blocked these guys through my firewall, and through my .htaccess however somehow they keep coming through. I just dont understand how they can do this?

Any thoughts?

Cheers.

1:03 pm on May 3, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 19, 2005
posts:11
votes: 0


Can someone explain what aipbot/ nameprotect is? hmmm rephrase - why it needs blocking? It seems to go through my site a few times a week and I was just trying to research it when I found this thread
1:07 pm on May 3, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


Their site is pretty clear:

[nameprotect.com...]

Their bot scans your site for copyright infractions related to their clients.

1:10 pm on May 3, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 19, 2005
posts:11
votes: 0


ok, so basically like a spy/ detective? Thanks
7:05 pm on May 4, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


It's been my experience that they try each UA on your site, if they are blocked they switch to a generic Windows + IE UA, so you have to resort to blocking their ranges.
2:13 am on May 5, 2005 (gmt 0)

New User

10+ Year Member

joined:Feb 29, 2004
posts:17
votes: 0


aipbot seems to have started to take a "if I behave I'll be let back in" approach in the last week or so. Previously it would read robots.txt and ignore it, persevering for a few 403'd pages. But now it just reads robots.txt, in which it is excluded.
8:02 pm on May 9, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 19, 2003
posts:695
votes: 0


Has anyone seen this?

24.177.134.6 - - [07/May/2005:03:08:02 -0400] "GET /robots.txt HTTP/1.0" 403 4429 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"

202.96.140.119 - - [07/May/2005:04:14:44 -0400] "GET / HTTP/1.0" 403 4429 "-" "AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , China)"

I'm confused. Ban both blocks? Are both scrapers, or what ever you want to call them, et. al.

1:18 pm on May 16, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:May 14, 2003
posts:164
votes: 0


The top one (24.177.134.6) has been battering my site on occasion for over a week. I set up a deny from today and the server load went from an average of 3.5 back down to 0.6.

Whether it's legitimate or not it was causing slowness on my server.

The page linked in the UA doesn't give any useful information, aside from what to put in the robots.txt file. No way to contact them to ask them to hold back on the page grabbing.

4:36 pm on May 16, 2005 (gmt 0)

New User

10+ Year Member

joined:Feb 29, 2004
posts:17
votes: 0


donovanh,

The IP you mention IS another manifestation of Nameprotect. It hit me first with the UA NPbot (Nameprotect's bot) then as aipbot a minute later and has stayed the same ever since.

Nameprotect seem to be widely disliked for excessive spidering, and I assume the use of another UA and IP is to get round those webmasters who've banned them on their 'official' IP/UA. They also use UA's from standard browsers from time to time on the IP's mentioned in in this thread, and I suppose they may well use others.

9:00 pm on May 17, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:May 14, 2003
posts:164
votes: 0


I sent an email to the e-mail address in the UA asking them to slow it down, but the e-mail has been delayed and doesn't seem to have been delivered.

It tried to grab 17,000 pages in a couple of hours, not enough to push it over, but quite a noticable strain on the server.

I'll be keeping an eye out for other manifestations.

9:23 pm on May 25, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 7, 2001
posts:608
votes: 0


These guys are not playing nice with me either.

I had

User-agent: nameprotect
Disallow: /

in my robots. then i went to NPBot. Now they are showing up in my logs as:

c24.177.134.6.static.ncr.charter.com

They just keep remorphing themselves so they can ignore the robots file. I'm gonna ban the whole ip range.

2:34 pm on May 26, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


> They just keep remorphing themselves so they can ignore the robots file.

Robots.txt compliance is optional in any case. They morph so you'll have difficulty banning their user agent via Mod_Rewrite or a straight deny from .htaccess ban.

> I'm gonna ban the whole ip range.

The only effective solution for now... until they lease new IP addresses.

3:16 am on May 29, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 20, 2004
posts:880
votes: 2


These jerks have sent bogus DMCA notices to many Amazon associates. I strongly urge everyone to not only block their access, but when they go around the blocks send them a cease and desist letter.
4:31 am on May 29, 2005 (gmt 0)

New User

10+ Year Member

joined:Feb 29, 2004
posts:17
votes: 0


These jerks have sent bogus DMCA notices to many Amazon associates.

Whats in it for them? Touting for business?

5:03 am on May 29, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


I know what Classless Inter-Domain Routing (CIDR) is, but despite reading several tutorials I still don't understand how to translate them to a network range.

For example, I want to block the 69.8.156.192/26 CIDR for NameProtect. Can this be translated to a traditional IP Address/Subnet Mask?

Thanks. :)

5:18 am on May 29, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Typed this up the other day. It's a reverse lookup on what you need, but maybe it'll help.

Subtract the start address of the range from the end address, keeping each dot-separated-column separate.

Look up the Netmask or CIDR based on this Difference.


Difference Addresses CIDR Netmask
0. 0. 0. 0 1 32 255.255.255.255
0. 0. 0. 1 2 31 255.255.255.254
0. 0. 0. 3 4 30 255.255.255.252
0. 0. 0. 7 8 29 255.255.255.248
0. 0. 0. 15 16 28 255.255.255.240
0. 0. 0. 31 32 27 255.255.255.224
0. 0. 0. 63 64 26 255.255.255.192
0. 0. 0.127 128 25 255.255.255.128
0. 0. 0.255 256 24 255.255.255.0

0. 0. 1.255 512 23 255.255.254.0
0. 0. 3.255 1,024 22 255.255.252.0
0. 0. 7.255 2,048 21 255.255.248.0
0. 0. 15.255 4,096 20 255.255.240.0
0. 0. 31.255 8,192 19 255.255.224.0
0. 0. 63.255 16,383 18 255.255.192.0
0. 0.127.255 32,767 17 255.255.128.0
0. 0.255.255 65,535 16 255.255.0.0

0. 1.255.255 131,072 15 255.254.0.0
0. 3.255.255 262,144 14 255.252.0.0
0. 7.255.255 524,288 13 255.248.0.0
0. 15.255.255 1,048,576 12 255.240.0.0
0. 31.255.255 2,097,152 11 255.224.0.0
0. 63.255.255 4,194,304 10 255.192.0.0
0.127.255.255 8,388,608 9 255.128.0.0
0.255.255.255 16,777,216 8 255.0.0.0

1.255.255.255 33,554,432 7 254.0.0.0
3.255.255.255 67,108,864 6 242.0.0.0
7.255.255.255 134,217,728 5 248.0.0.0
15.255.255.255 268,435,456 4 240.0.0.0
31.255.255.255 536,870,912 3 224.0.0.0
63.255.255.255 1,073,741,824 2 192.0.0.0
127.255.255.255 2,147,483,648 1 128.0.0.0
255.255.255.255 4,294,967,296 0 0.0.0.0.0

Hopefully, the formatting won't be too messed-up by posting.

You might also try a search for CIDR Network Netmask converter... There are several out there.

Jim

6:22 am on May 29, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


Thanks Jim.

The formatting was awful. Or maybe it was the Rum. :)

I found a freeware calculator. I don't think I understand it though.

Clearly I'm off-topic and I appreciate your indulgence but I'll pick a more appropriate forum to ask for more help.

12:54 pm on May 29, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


For example, I want to block the 69.8.156.192/26 CIDR for NameProtect. Can this be translated to a traditional IP Address/Subnet Mask?

This page has a free conversion tool.
Second column, 2nd last box near bottom.
[dnsstuff.com...]

provides the following:
[dnsstuff.com...]

3:55 am on May 30, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


Great site. Thanks wilderness.
3:26 am on June 2, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Dec 20, 2003
posts:268
votes: 0


Sounds like you have the mask aready, but in case... a /26 cidr notation corresponds to a 255.255.255.192 subnet mask.

All commercial spiders utilize your commercial resources for their commercial benefit. With the major search engines it is mostly win-win because they generate trafic in return. With many other commercial bots there is no upside. I always ban commercial bots where there is no upside without particular regard to what their commercial interests are.

4:25 am on June 2, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


Thank you, idoc.

I know about user agents being as I publish the web's only regularly updated browscap.ini file.

What I'm not so good at is the technical side of things like translating a CIDR into something my software can make use of. :)

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members