homepage Welcome to WebmasterWorld Guest from 54.166.255.168
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
abot/0.1
New name for npbot.
fiestagirl

10+ Year Member



 
Msg#: 2799 posted 6:31 pm on Mar 31, 2005 (gmt 0)

Old ua:
np/0.1_(np;_http://www.nameprotect.com;_npbot@nameprotect.com)

New name:
abot/0.1_(abot;_http://www.abot.com;_abot@abot.com)

IP: 24.177.134.6

 

volatilegx

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 7:05 pm on Mar 31, 2005 (gmt 0)

Very nice catch, fiestagirl!

jmccormac

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



 
Msg#: 2799 posted 7:33 pm on Mar 31, 2005 (gmt 0)

Yep - the gobsh1tes in NP decided to hit my site fairly hard this morning. (It is the main reference site on Irish (.ie) domain names.) I had to add a permanent IP ban. I was wondering who abot.com was.A check on the abot.com hoster from their domain name shows 4851 domains registered as of March 01. The nameprotect.com hoster shows 2010 domains hosted. An IP level ban is so much more efficient than having to ban each potential spoof incarnation individually.

Regards...jmcc

rfontaine

10+ Year Member



 
Msg#: 2799 posted 6:13 pm on Apr 1, 2005 (gmt 0)

hummm....

Deny from 24.177.134.6

doesn't stop them.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 2799 posted 7:09 pm on Apr 1, 2005 (gmt 0)

try deny from 24.177.
or
RewriteCond %{REMOTE_ADDR} ^24\.177\.1(2[89]¦3[0-5])\. [OR]

the deny you used with the last Class of (6) is both too narrow and specific.

idoc

10+ Year Member



 
Msg#: 2799 posted 11:04 pm on Apr 1, 2005 (gmt 0)

or:
Deny from 24.177.128.0/21

pendanticist

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 1:04 pm on Apr 2, 2005 (gmt 0)

24.177.134.6 - - [01/Apr/2005:22:15:42 -0800] "GET /robots.txt HTTP/1.0" 200 2200 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"
24.177.134.6 - - [01/Apr/2005:22:15:43 -0800] "GET / HTTP/1.0" 403 480 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"
24.177.134.6 - - [01/Apr/2005:22:26:08 -0800] "GET /Blah.html HTTP/1.0" 403 480 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"

A rose by any other name...

carrot63

10+ Year Member



 
Msg#: 2799 posted 5:28 am on Apr 16, 2005 (gmt 0)

Hi all,

The CIDR ranges and single IP below cover everything I've found for Nameprotect / NP / NPbot / aibot / abot. If anyone has any more let me know.

24.177.134.6
12.175.0.32/28
12.148.209.192/26
12.148.196.128/25
69.8.156.192/26

JohnCanyon

10+ Year Member



 
Msg#: 2799 posted 5:56 pm on Apr 27, 2005 (gmt 0)

I have blocked these guys through my firewall, and through my .htaccess however somehow they keep coming through. I just dont understand how they can do this?

Any thoughts?

Cheers.

themetalpeddler

5+ Year Member



 
Msg#: 2799 posted 1:03 pm on May 3, 2005 (gmt 0)

Can someone explain what aipbot/ nameprotect is? hmmm rephrase - why it needs blocking? It seems to go through my site a few times a week and I was just trying to research it when I found this thread

bcolflesh

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 1:07 pm on May 3, 2005 (gmt 0)

Their site is pretty clear:

[nameprotect.com...]

Their bot scans your site for copyright infractions related to their clients.

themetalpeddler

5+ Year Member



 
Msg#: 2799 posted 1:10 pm on May 3, 2005 (gmt 0)

ok, so basically like a spy/ detective? Thanks

bcolflesh

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 7:05 pm on May 4, 2005 (gmt 0)

It's been my experience that they try each UA on your site, if they are blocked they switch to a generic Windows + IE UA, so you have to resort to blocking their ranges.

carrot63

10+ Year Member



 
Msg#: 2799 posted 2:13 am on May 5, 2005 (gmt 0)

aipbot seems to have started to take a "if I behave I'll be let back in" approach in the last week or so. Previously it would read robots.txt and ignore it, persevering for a few 403'd pages. But now it just reads robots.txt, in which it is excluded.

jim_w

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 8:02 pm on May 9, 2005 (gmt 0)

Has anyone seen this?

24.177.134.6 - - [07/May/2005:03:08:02 -0400] "GET /robots.txt HTTP/1.0" 403 4429 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"

202.96.140.119 - - [07/May/2005:04:14:44 -0400] "GET / HTTP/1.0" 403 4429 "-" "AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , China)"

I'm confused. Ban both blocks? Are both scrapers, or what ever you want to call them, et. al.

donovanh

10+ Year Member



 
Msg#: 2799 posted 1:18 pm on May 16, 2005 (gmt 0)

The top one (24.177.134.6) has been battering my site on occasion for over a week. I set up a deny from today and the server load went from an average of 3.5 back down to 0.6.

Whether it's legitimate or not it was causing slowness on my server.

The page linked in the UA doesn't give any useful information, aside from what to put in the robots.txt file. No way to contact them to ask them to hold back on the page grabbing.

carrot63

10+ Year Member



 
Msg#: 2799 posted 4:36 pm on May 16, 2005 (gmt 0)

donovanh,

The IP you mention IS another manifestation of Nameprotect. It hit me first with the UA NPbot (Nameprotect's bot) then as aipbot a minute later and has stayed the same ever since.

Nameprotect seem to be widely disliked for excessive spidering, and I assume the use of another UA and IP is to get round those webmasters who've banned them on their 'official' IP/UA. They also use UA's from standard browsers from time to time on the IP's mentioned in in this thread, and I suppose they may well use others.

donovanh

10+ Year Member



 
Msg#: 2799 posted 9:00 pm on May 17, 2005 (gmt 0)

I sent an email to the e-mail address in the UA asking them to slow it down, but the e-mail has been delayed and doesn't seem to have been delivered.

It tried to grab 17,000 pages in a couple of hours, not enough to push it over, but quite a noticable strain on the server.

I'll be keeping an eye out for other manifestations.

wingslevel

10+ Year Member



 
Msg#: 2799 posted 9:23 pm on May 25, 2005 (gmt 0)

These guys are not playing nice with me either.

I had

User-agent: nameprotect
Disallow: /

in my robots. then i went to NPBot. Now they are showing up in my logs as:

c24.177.134.6.static.ncr.charter.com

They just keep remorphing themselves so they can ignore the robots file. I'm gonna ban the whole ip range.

volatilegx

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 2:34 pm on May 26, 2005 (gmt 0)

> They just keep remorphing themselves so they can ignore the robots file.

Robots.txt compliance is optional in any case. They morph so you'll have difficulty banning their user agent via Mod_Rewrite or a straight deny from .htaccess ban.

> I'm gonna ban the whole ip range.

The only effective solution for now... until they lease new IP addresses.

eeek

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 3:16 am on May 29, 2005 (gmt 0)

These jerks have sent bogus DMCA notices to many Amazon associates. I strongly urge everyone to not only block their access, but when they go around the blocks send them a cease and desist letter.

carrot63

10+ Year Member



 
Msg#: 2799 posted 4:31 am on May 29, 2005 (gmt 0)

These jerks have sent bogus DMCA notices to many Amazon associates.

Whats in it for them? Touting for business?

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 5:03 am on May 29, 2005 (gmt 0)

I know what Classless Inter-Domain Routing (CIDR) is, but despite reading several tutorials I still don't understand how to translate them to a network range.

For example, I want to block the 69.8.156.192/26 CIDR for NameProtect. Can this be translated to a traditional IP Address/Subnet Mask?

Thanks. :)

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2799 posted 5:18 am on May 29, 2005 (gmt 0)

Typed this up the other day. It's a reverse lookup on what you need, but maybe it'll help.

Subtract the start address of the range from the end address, keeping each dot-separated-column separate.

Look up the Netmask or CIDR based on this Difference.


Difference Addresses CIDR Netmask
0. 0. 0. 0 1 32 255.255.255.255
0. 0. 0. 1 2 31 255.255.255.254
0. 0. 0. 3 4 30 255.255.255.252
0. 0. 0. 7 8 29 255.255.255.248
0. 0. 0. 15 16 28 255.255.255.240
0. 0. 0. 31 32 27 255.255.255.224
0. 0. 0. 63 64 26 255.255.255.192
0. 0. 0.127 128 25 255.255.255.128
0. 0. 0.255 256 24 255.255.255.0

0. 0. 1.255 512 23 255.255.254.0
0. 0. 3.255 1,024 22 255.255.252.0
0. 0. 7.255 2,048 21 255.255.248.0
0. 0. 15.255 4,096 20 255.255.240.0
0. 0. 31.255 8,192 19 255.255.224.0
0. 0. 63.255 16,383 18 255.255.192.0
0. 0.127.255 32,767 17 255.255.128.0
0. 0.255.255 65,535 16 255.255.0.0

0. 1.255.255 131,072 15 255.254.0.0
0. 3.255.255 262,144 14 255.252.0.0
0. 7.255.255 524,288 13 255.248.0.0
0. 15.255.255 1,048,576 12 255.240.0.0
0. 31.255.255 2,097,152 11 255.224.0.0
0. 63.255.255 4,194,304 10 255.192.0.0
0.127.255.255 8,388,608 9 255.128.0.0
0.255.255.255 16,777,216 8 255.0.0.0

1.255.255.255 33,554,432 7 254.0.0.0
3.255.255.255 67,108,864 6 242.0.0.0
7.255.255.255 134,217,728 5 248.0.0.0
15.255.255.255 268,435,456 4 240.0.0.0
31.255.255.255 536,870,912 3 224.0.0.0
63.255.255.255 1,073,741,824 2 192.0.0.0
127.255.255.255 2,147,483,648 1 128.0.0.0
255.255.255.255 4,294,967,296 0 0.0.0.0.0

Hopefully, the formatting won't be too messed-up by posting.

You might also try a search for CIDR Network Netmask converter... There are several out there.

Jim

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 6:22 am on May 29, 2005 (gmt 0)

Thanks Jim.

The formatting was awful. Or maybe it was the Rum. :)

I found a freeware calculator. I don't think I understand it though.

Clearly I'm off-topic and I appreciate your indulgence but I'll pick a more appropriate forum to ask for more help.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 2799 posted 12:54 pm on May 29, 2005 (gmt 0)

For example, I want to block the 69.8.156.192/26 CIDR for NameProtect. Can this be translated to a traditional IP Address/Subnet Mask?

This page has a free conversion tool.
Second column, 2nd last box near bottom.
[dnsstuff.com...]

provides the following:
[dnsstuff.com...]

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 3:55 am on May 30, 2005 (gmt 0)

Great site. Thanks wilderness.

idoc

10+ Year Member



 
Msg#: 2799 posted 3:26 am on Jun 2, 2005 (gmt 0)

Sounds like you have the mask aready, but in case... a /26 cidr notation corresponds to a 255.255.255.192 subnet mask.

All commercial spiders utilize your commercial resources for their commercial benefit. With the major search engines it is mostly win-win because they generate trafic in return. With many other commercial bots there is no upside. I always ban commercial bots where there is no upside without particular regard to what their commercial interests are.

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2799 posted 4:25 am on Jun 2, 2005 (gmt 0)

Thank you, idoc.

I know about user agents being as I publish the web's only regularly updated browscap.ini file.

What I'm not so good at is the technical side of things like translating a CIDR into something my software can make use of. :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved