homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

New name for npbot.

 6:31 pm on Mar 31, 2005 (gmt 0)

Old ua:

New name:




 7:05 pm on Mar 31, 2005 (gmt 0)

Very nice catch, fiestagirl!


 7:33 pm on Mar 31, 2005 (gmt 0)

Yep - the gobsh1tes in NP decided to hit my site fairly hard this morning. (It is the main reference site on Irish (.ie) domain names.) I had to add a permanent IP ban. I was wondering who abot.com was.A check on the abot.com hoster from their domain name shows 4851 domains registered as of March 01. The nameprotect.com hoster shows 2010 domains hosted. An IP level ban is so much more efficient than having to ban each potential spoof incarnation individually.



 6:13 pm on Apr 1, 2005 (gmt 0)


Deny from

doesn't stop them.


 7:09 pm on Apr 1, 2005 (gmt 0)

try deny from 24.177.
RewriteCond %{REMOTE_ADDR} ^24\.177\.1(2[89]¦3[0-5])\. [OR]

the deny you used with the last Class of (6) is both too narrow and specific.


 11:04 pm on Apr 1, 2005 (gmt 0)

Deny from


 1:04 pm on Apr 2, 2005 (gmt 0) - - [01/Apr/2005:22:15:42 -0800] "GET /robots.txt HTTP/1.0" 200 2200 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)" - - [01/Apr/2005:22:15:43 -0800] "GET / HTTP/1.0" 403 480 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)" - - [01/Apr/2005:22:26:08 -0800] "GET /Blah.html HTTP/1.0" 403 480 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"

A rose by any other name...


 5:28 am on Apr 16, 2005 (gmt 0)

Hi all,

The CIDR ranges and single IP below cover everything I've found for Nameprotect / NP / NPbot / aibot / abot. If anyone has any more let me know.


 5:56 pm on Apr 27, 2005 (gmt 0)

I have blocked these guys through my firewall, and through my .htaccess however somehow they keep coming through. I just dont understand how they can do this?

Any thoughts?



 1:03 pm on May 3, 2005 (gmt 0)

Can someone explain what aipbot/ nameprotect is? hmmm rephrase - why it needs blocking? It seems to go through my site a few times a week and I was just trying to research it when I found this thread


 1:07 pm on May 3, 2005 (gmt 0)

Their site is pretty clear:


Their bot scans your site for copyright infractions related to their clients.


 1:10 pm on May 3, 2005 (gmt 0)

ok, so basically like a spy/ detective? Thanks


 7:05 pm on May 4, 2005 (gmt 0)

It's been my experience that they try each UA on your site, if they are blocked they switch to a generic Windows + IE UA, so you have to resort to blocking their ranges.


 2:13 am on May 5, 2005 (gmt 0)

aipbot seems to have started to take a "if I behave I'll be let back in" approach in the last week or so. Previously it would read robots.txt and ignore it, persevering for a few 403'd pages. But now it just reads robots.txt, in which it is excluded.


 8:02 pm on May 9, 2005 (gmt 0)

Has anyone seen this? - - [07/May/2005:03:08:02 -0400] "GET /robots.txt HTTP/1.0" 403 4429 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)" - - [07/May/2005:04:14:44 -0400] "GET / HTTP/1.0" 403 4429 "-" "AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , China)"

I'm confused. Ban both blocks? Are both scrapers, or what ever you want to call them, et. al.


 1:18 pm on May 16, 2005 (gmt 0)

The top one ( has been battering my site on occasion for over a week. I set up a deny from today and the server load went from an average of 3.5 back down to 0.6.

Whether it's legitimate or not it was causing slowness on my server.

The page linked in the UA doesn't give any useful information, aside from what to put in the robots.txt file. No way to contact them to ask them to hold back on the page grabbing.


 4:36 pm on May 16, 2005 (gmt 0)


The IP you mention IS another manifestation of Nameprotect. It hit me first with the UA NPbot (Nameprotect's bot) then as aipbot a minute later and has stayed the same ever since.

Nameprotect seem to be widely disliked for excessive spidering, and I assume the use of another UA and IP is to get round those webmasters who've banned them on their 'official' IP/UA. They also use UA's from standard browsers from time to time on the IP's mentioned in in this thread, and I suppose they may well use others.


 9:00 pm on May 17, 2005 (gmt 0)

I sent an email to the e-mail address in the UA asking them to slow it down, but the e-mail has been delayed and doesn't seem to have been delivered.

It tried to grab 17,000 pages in a couple of hours, not enough to push it over, but quite a noticable strain on the server.

I'll be keeping an eye out for other manifestations.


 9:23 pm on May 25, 2005 (gmt 0)

These guys are not playing nice with me either.

I had

User-agent: nameprotect
Disallow: /

in my robots. then i went to NPBot. Now they are showing up in my logs as:


They just keep remorphing themselves so they can ignore the robots file. I'm gonna ban the whole ip range.


 2:34 pm on May 26, 2005 (gmt 0)

> They just keep remorphing themselves so they can ignore the robots file.

Robots.txt compliance is optional in any case. They morph so you'll have difficulty banning their user agent via Mod_Rewrite or a straight deny from .htaccess ban.

> I'm gonna ban the whole ip range.

The only effective solution for now... until they lease new IP addresses.


 3:16 am on May 29, 2005 (gmt 0)

These jerks have sent bogus DMCA notices to many Amazon associates. I strongly urge everyone to not only block their access, but when they go around the blocks send them a cease and desist letter.


 4:31 am on May 29, 2005 (gmt 0)

These jerks have sent bogus DMCA notices to many Amazon associates.

Whats in it for them? Touting for business?


 5:03 am on May 29, 2005 (gmt 0)

I know what Classless Inter-Domain Routing (CIDR) is, but despite reading several tutorials I still don't understand how to translate them to a network range.

For example, I want to block the CIDR for NameProtect. Can this be translated to a traditional IP Address/Subnet Mask?

Thanks. :)


 5:18 am on May 29, 2005 (gmt 0)

Typed this up the other day. It's a reverse lookup on what you need, but maybe it'll help.

Subtract the start address of the range from the end address, keeping each dot-separated-column separate.

Look up the Netmask or CIDR based on this Difference.

Difference Addresses CIDR Netmask
0. 0. 0. 0 1 32
0. 0. 0. 1 2 31
0. 0. 0. 3 4 30
0. 0. 0. 7 8 29
0. 0. 0. 15 16 28
0. 0. 0. 31 32 27
0. 0. 0. 63 64 26
0. 0. 0.127 128 25
0. 0. 0.255 256 24

0. 0. 1.255 512 23
0. 0. 3.255 1,024 22
0. 0. 7.255 2,048 21
0. 0. 15.255 4,096 20
0. 0. 31.255 8,192 19
0. 0. 63.255 16,383 18
0. 0.127.255 32,767 17
0. 0.255.255 65,535 16

0. 1.255.255 131,072 15
0. 3.255.255 262,144 14
0. 7.255.255 524,288 13
0. 15.255.255 1,048,576 12
0. 31.255.255 2,097,152 11
0. 63.255.255 4,194,304 10 8,388,608 9 16,777,216 8 33,554,432 7 67,108,864 6 134,217,728 5 268,435,456 4 536,870,912 3 1,073,741,824 2 2,147,483,648 1 4,294,967,296 0

Hopefully, the formatting won't be too messed-up by posting.

You might also try a search for CIDR Network Netmask converter... There are several out there.



 6:22 am on May 29, 2005 (gmt 0)

Thanks Jim.

The formatting was awful. Or maybe it was the Rum. :)

I found a freeware calculator. I don't think I understand it though.

Clearly I'm off-topic and I appreciate your indulgence but I'll pick a more appropriate forum to ask for more help.


 12:54 pm on May 29, 2005 (gmt 0)

For example, I want to block the CIDR for NameProtect. Can this be translated to a traditional IP Address/Subnet Mask?

This page has a free conversion tool.
Second column, 2nd last box near bottom.

provides the following:


 3:55 am on May 30, 2005 (gmt 0)

Great site. Thanks wilderness.


 3:26 am on Jun 2, 2005 (gmt 0)

Sounds like you have the mask aready, but in case... a /26 cidr notation corresponds to a subnet mask.

All commercial spiders utilize your commercial resources for their commercial benefit. With the major search engines it is mostly win-win because they generate trafic in return. With many other commercial bots there is no upside. I always ban commercial bots where there is no upside without particular regard to what their commercial interests are.


 4:25 am on Jun 2, 2005 (gmt 0)

Thank you, idoc.

I know about user agents being as I publish the web's only regularly updated browscap.ini file.

What I'm not so good at is the technical side of things like translating a CIDR into something my software can make use of. :)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved