Welcome to WebmasterWorld Guest from 107.20.75.63

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

IP Lists

     
4:32 pm on Jun 10, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Ran into this noncommercial site with free ip lists to the major search engines. This is something many of us have wanted for years:
[iplists.com...]
4:54 pm on June 10, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member nick_w is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Feb 4, 2002
posts:5044
votes: 0


Great find!

Wonder how accurate it is?

Nick

5:07 pm on June 10, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Oh, it's very accurate.
5:12 pm on June 10, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


But a bit dated.
Still a very useful list.
5:14 pm on June 10, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member nick_w is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Feb 4, 2002
posts:5044
votes: 0


Maybe the IP's just have not altered since the date on the last revision?

How would you get these details, you'd have to see the IP's hit your site right?

Nick

5:32 pm on June 10, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


Hi Guys...

It's my site/list ... questions? Comments?

It's as up to date and accurate as I can keep it. If anybody has IPs that I don't... I'd certainly appreciate knowing about them. I post many new ones that I find here and also glean quite a bit of info here, too, as many of you already know.

Thanks!

<added>Oh yeah, there is a rearranged version located here: [iplists.com...] . The rearranged version reflects some shifts in Search Engine prominence.</added>

[edited by: volatilegx at 3:04 pm (utc) on Oct. 25, 2004]

ann

9:28 pm on June 10, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 25, 2002
posts:2605
votes: 0


Does MSN have a spider? Looked at my logs today and msn was all over them????

Ann

10:13 pm on June 10, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


I belive MSN uses the Inktomi index... can you give some examples of the log entries with MSN in them?
10:38 pm on June 10, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 31, 2001
posts:1238
votes: 0


Nice find Brett! I've got a new toy... AND a very useful one!
11:09 pm on June 10, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 24, 2001
posts:501
votes: 0


I've had 2 major sites trashed in the last 3 months owing to cloaking with outdated IP lists. Lists from an expensive service that were always current but outdated immediately when new spider IPs were introduced.

I asked one customer of mine if they were going to fire me because of it. They laughed. It seems I cost them so much money they couldn't afford to fire me. I'll be working it off with them over the next 6-8 years.

Job security in it's basic form.

11:23 pm on June 10, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


>>>Lists from an expensive service that were always current but outdated immediately when new spider IPs were introduced.

You mean I can make money off of IP lists? ;) I don't cloak but I knew there was a reason why I've been hoarding those IP lists away all this time. If you don't mind, I would be interested in knowing the name of the service you were dealing with. You can stickymail it you like.

7:03 am on June 11, 2002 (gmt 0)

Full Member

10+ Year Member

joined:June 10, 2002
posts:203
votes: 0


Great resource volatilegx!

Would love to see a search for ip option on your site. btw, I was hitted by 209.10.169.24 and 64.140.48.38 with 10K hits each! , are these IP belongs to SE? Whis information below:
Thanks
Hanuman

209.10.169.24
^^^^^^^^^^^^^
Globix Corporation (NETBLK-GLOBIXBLK3)
295 Lafayette St- 3rd Fl

NY, NY 10012

US

Netname: GLOBIXBLK3
Netblock: 209.10.0.0 - 209.11.223.255
Maintainer: PFMC

Coordinator:
Hostmaster, Globix Corporation (GCH2-ARIN) arin-admin@GLOBIX.NET
+1-212-334-8500 (FAX) 212.334.8615

Domain System inverse mapping provided by:

Z1.NS.NYC1.GLOBIX.NET 209.10.66.55
Z1.NS.SJC1.GLOBIX.NET 209.10.34.55
Z1.NS.LHR1.GLOBIX.NET 212.111.32.38

64.140.48.30
^^^^^^^^^^^^
ICG NetAhead, Inc. (NETBLK-ICG-BLK5)
161 Inverness Dr. West

Englewood, CO 80112

US

Netname: ICG-BLK5
Netblock: 64.140.0.0 - 64.140.95.255
Maintainer: ICGN

Coordinator:
Taylor, Stacy (ST452-ARIN) abuse@icgcom.com
408-579-5000

Domain System inverse mapping provided by:

AS1.ICG.NET 170.147.45.163
AS2.ICG.NET 170.147.45.164

7:11 am on June 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


Yeah, volatilegx, it is really good of you to make this publicly available.
4:45 pm on June 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


hanuman,

I'm following up with the network owners of those IP addresses. Hopefully, I'll get some information. I'm especially curious about the bot out of Colorado, as I've seen a number of unidentified bots coming out of Englewood, CO.

I'll post what I find here.

4:58 pm on June 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2001
posts:1609
votes: 0


Thanks volatilegx !
5:27 pm on June 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


I've added sort of a crude search feature. [iplists.com...]

You can input a list of IP Addresses, one per line, and you'll be told if they were found in my lists, and which list they were on.

10:17 pm on June 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


hanuman,

64.140.48.30, the IP from Colorado, appears to belong to the company Iparadigms.com. They are a copyright violation search company. They appear to be related to TurnItIn.com, Plagairism.org and SlySearch. Prime banning material and definitely not a search engine. Did you happen to get the User Agent on this one? I'm betting it was the SlySearch bot. According to the documentation for the SlySearch bot, it obeys robots.txt. Docs: [slysearch.com...]

Still working on the other one.

10:28 pm on June 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


hanuman,

209.10.169.24 is most likely PortalBSpider/2.0 (spider@portalb.com)

[portalb.com...]

Hope this helps.

tvr23

1:33 pm on June 12, 2002 (gmt 0)

Inactive Member
Account Expired

 
 


Not sure if this is the correct place to post this?

I am currently working on a project where I need to identfy major ISP providers from IP's - much like the IPlists.com site identifies the search engines.

The ISP's i need to find are aol, bt open world, freeserve, pippex and compuserve. Are there any sites that would give me this information?

Thanks in advance

3:36 pm on June 12, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 17, 2002
posts:86
votes: 0


Call me stupid, but what is the benefit of having the Search Engines IP? Would it be to see their visits in your stats? Going one step further it would be cool to have a stats package that highlighted the Hits/IP's that came from search engines.
3:41 pm on June 12, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


One reason to collect search engine IP addresses is for cloaking purposes.

Another is to make sure you don't ban the search engine IP addresses from your website. Many people ban other bots but want to leave the SE IPs alone.

4:03 pm on June 12, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5408
votes: 2


<snip>Going one step further it would be cool to have a stats package that highlighted the Hits/IP's that came from search engines>

Analog does an excellent job of accumulating search engine stats. Although it doesn't highlight them.
Analog does most anything if you can figure out how to configure it.

4:06 pm on June 12, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 17, 2002
posts:86
votes: 0


hmm cloaking. is that similar to taking the invisible ink and painting yourself with? could you point me somewhere or just give me a gen. description please?
4:10 pm on June 12, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member nffc is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 22, 2000
posts:3604
votes: 0


[searchengineworld.com...] is a good starting point OhMyPixel.
4:13 pm on June 12, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


Also look here: [webmasterworld.com...]
4:36 pm on June 12, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 17, 2002
posts:86
votes: 0


thank you, reading it now.
1:41 am on June 13, 2002 (gmt 0)

Full Member

10+ Year Member

joined:June 10, 2002
posts:203
votes: 0


Thank you volatilegx for the help in the matter! and Key_Master for the information.!

Hanuman

4:47 am on June 13, 2002 (gmt 0)

Full Member

10+ Year Member

joined:June 10, 2002
posts:203
votes: 0


To block 209.10.169.24 - PortalBSpider, and 64.140.48.30 Slysearch

I added these lines to my .htaccess file

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^(SlySearch.*¦PortalBSpider.*) [NC,OR]
RewriteRule ^(.*) block.htm [L]

I would also recommend adding the following lines

RewriteCond %{HTTP_USER_AGENT} ^(-?¦[A-Z]{10})$ [OR]

RewriteCond %{REMOTE_HOST} ^private$ [NC,OR]

Thanks the group for the kind help!
Hanuman

7:09 pm on June 13, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 20, 2002
posts:148
votes: 0


Thanks volatilegx!

Man, that's a LOT of IPs!

Many more than I'm currently using.

Is the old list obsolete and can be ignored or should it be cm0bined with the new?

Also: Like I said, that's a LOT of IPs. Are your sure none of these are non SE spiders. I mean I've had clients get really pissed when one of their memebrs accessed cloaked pages because I included a block of IPs that contained one their ISP was using.

:-(

7:13 pm on June 13, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 20, 2002
posts:148
votes: 0


Regarding my above post:

I noticed some IPs in the "New" list that were included as entire blocks in the "Old" list.

Does this mean you discovered that the entire block was not correct?

Thanks Again,
Friday

This 36 message thread spans 2 pages: 36