Looking for data on internet filters

[volatilegx]

I'm looking to make a new section of iplists.com devoted to tracking internet filter spiders. I don't have much data to work with at the moment, so I would appreciate anything you guys have :)

Thanks,

Dan

[Intrpngn]

Here is my list from kloth plus some additional bots

#Kill bad bots that just do not listen (http://www.kloth.net/internet/badbots.php)
RewriteCond %{REMOTE_ADDR} ^211\.147\.3\.21$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.95\.23\.180$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.95\.23\.181$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.95\.23\.184$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.106\.127\.199$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.101\.4\.16$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.101\.42$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.101\.42$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.108\.221\.173$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.108\.221\.97$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.181\.70\.86$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.160\.72\.53$ [OR]
RewriteCond %{REMOTE_ADDR} ^207\.31\.251\.130$ [OR]
RewriteCond %{REMOTE_ADDR} ^207\.31\.249\.213$ [OR]
RewriteCond %{REMOTE_ADDR} ^207\.31\.251\.150$ [OR]
RewriteCond %{REMOTE_ADDR} ^207\.31\.249\.217$ [OR]
RewriteCond %{REMOTE_ADDR} ^207\.31\.251\.146$ [OR]
RewriteCond %{REMOTE_ADDR} ^220\.73\.165\.141$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.51\.1\.51$ [OR]
RewriteCond %{REMOTE_ADDR} ^210\.59\.144\.148$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.65\.80\.221$ [OR]
RewriteCond %{REMOTE_ADDR} ^199\.231\.148\.148$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.205\.220\.2$ [OR]
RewriteCond %{REMOTE_ADDR} ^213\.213\.141\.250$ [OR]
RewriteCond %{REMOTE_ADDR} ^62\.103\.197\.40$ [OR]
RewriteCond %{REMOTE_ADDR} ^204\.210\.156\.14$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.36\.6\.250$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.28\.23\.147$ [OR]
RewriteCond %{REMOTE_ADDR} ^210\.182\.147\.184$ [OR]
RewriteCond %{REMOTE_ADDR} ^68\.142\.141\.220$ [OR]
RewriteCond %{REMOTE_ADDR} ^68\.14\.144\.194$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.8\.42$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.8\.41$ [OR]
RewriteCond %{REMOTE_ADDR} ^209\.25\.160\.100$ [OR]
RewriteCond %{REMOTE_ADDR} ^220\.130\.5\.7$ [OR]
RewriteCond %{REMOTE_ADDR} ^67\.19\.60\.58$ [OR]
RewriteCond %{REMOTE_ADDR} ^67\.18\.218\.122$ [OR]
RewriteCond %{REMOTE_ADDR} ^67\.18\.218\.226$ [OR]
RewriteCond %{REMOTE_ADDR} ^67\.18\.133\.10$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.208\.106\.78$ [OR]
RewriteCond %{REMOTE_ADDR} ^67\.138\.247\.2$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.237$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.246$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.250$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.247$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.253$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.229$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.233$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.233$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.232$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.106\.79\.73$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.45\.16\.199$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.189\.111\.254$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.95\.207\.247$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.118\.3\.227$ [OR]
RewriteCond %{REMOTE_ADDR} ^82\.195\.131\.177$ [OR]
RewriteCond %{REMOTE_ADDR} ^213\.215\.216\.13$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.79\.181\.80$ [OR]
RewriteCond %{REMOTE_ADDR} ^67\.71\.135\.181$ [OR]
RewriteCond %{REMOTE_ADDR} ^83\.116\.34\.185$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.162\.208\.27$ [OR]
RewriteCond %{REMOTE_ADDR} ^220\.114\.3\.9$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.8$ [OR]
RewriteCond %{REMOTE_ADDR} ^130\.230\.150\.113$ [OR]
RewriteCond %{REMOTE_ADDR} ^68\.110\.114\.196$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.110\.48\.143$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.194\.74\.66$ [OR]
RewriteCond %{REMOTE_ADDR} ^62\.220\.103\.6$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.55\.90\.173$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.251\.54\.207$ [OR]
RewriteCond %{REMOTE_ADDR} ^62\.194\.21\.21$ [OR]
RewriteCond %{REMOTE_ADDR} ^62\.194\.103\.4$ [OR]
RewriteCond %{REMOTE_ADDR} ^62\.194\.16\.57$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.43\.113\.169$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.69\.248\.64$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.246\.0\.17$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.164\.136\.220$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.98\.0\.23$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.103\.132\.121$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.207\.118\.206$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.152\.14\.95$ [OR]
RewriteCond %{REMOTE_ADDR} ^219\.68\.24\.39$ [OR]
RewriteCond %{REMOTE_ADDR} ^195\.166\.237\.40$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.179\.100\.106$ [OR]
RewriteCond %{REMOTE_ADDR} ^203\.101\.66\.14$ [OR]
RewriteCond %{REMOTE_ADDR} ^62\.163\.72\.90$ [OR]
RewriteCond %{REMOTE_ADDR} ^213\.206\.5\.5$ [OR]
RewriteCond %{REMOTE_ADDR} ^168\.75\.177\.2$ [OR]
RewriteCond %{REMOTE_ADDR} ^209\.247\.193\.211$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.69\.51\.217$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.15\.140\.146$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.143\.135\.135$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.81\.45\.168$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.66\.161\.192$ [OR]
RewriteCond %{REMOTE_ADDR} ^195\.127\.27\.228$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.36\.6$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.36\.9$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.36\.7$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.35\.145$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.56\.208\.114$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.199\.84\.12$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.199\.83\.15$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.18\.74\.97$ [OR]
RewriteCond %{REMOTE_ADDR} ^210\.206\.171\.79$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.19\.224\.180$ [OR]
RewriteCond %{REMOTE_ADDR} ^219\.93\.212\.58$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.110\.240\.195$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.52\.161\.83$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.30\.13\.176$ [OR]
RewriteCond %{REMOTE_ADDR} ^216\.205\.151\.228$ [OR]
RewriteCond %{REMOTE_ADDR} ^216\.205\.91\.230$ [OR]
RewriteCond %{REMOTE_ADDR} ^67\.10\.108\.42$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.161\.24\.2$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.6\.247\.154$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.99\.213\.19$ [OR]
RewriteCond %{REMOTE_ADDR} ^213\.215\.133\.19$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.56\.213\.178$ [OR]
RewriteCond %{REMOTE_ADDR} ^212\.55\.244\.74$ [OR]
RewriteCond %{REMOTE_ADDR} ^67\.39\.41\.147$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.17\.238\.156$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.17\.238\.156$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.173\.106\.90$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.134\.200\.78$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.15\.135\.123$ [OR]
RewriteCond %{REMOTE_ADDR} ^219\.129\.181\.72$ [OR]
RewriteCond %{REMOTE_ADDR} ^219\.129\.179\.32$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.15\.139\.93$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.102\.199\.195$ [OR]
RewriteCond %{REMOTE_ADDR} ^208\.135\.47\.2$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.99\.203\.199$ [OR]
RewriteCond %{REMOTE_ADDR} ^212\.98\.46\.104$ [OR]
RewriteCond %{REMOTE_ADDR} ^212\.98\.46\.104$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.161\.66\.143$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.35\.106\.66$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.218\.79\.184$ [OR]
RewriteCond %{REMOTE_ADDR} ^68\.104\.17\.97$ [OR]
RewriteCond %{REMOTE_ADDR} ^68\.104\.17\.97$ [OR]
RewriteCond %{REMOTE_ADDR} ^68\.104\.56\.9$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.56\.131\.218$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.207\.120\.227$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.51\.1\.51$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.50\.139\.130$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.121\.41\.174$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.28\.139\.66$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.173\.88\.244$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.66\.94\.141$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.69\.255\.237$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.70\.95\.204$ [OR]
RewriteCond %{REMOTE_ADDR} ^213\.249\.54\.203$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.129\.132\.144$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.129\.135\.132$ [OR]
RewriteCond %{REMOTE_ADDR} ^138\.15\.164\.16$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.13\.152\.85$ [OR]
RewriteCond %{REMOTE_ADDR} ^213\.249\.18\.67$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.252\.233\.116$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.99\.203\.214$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.114\.189\.44$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.92\.75\.22$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.226\.33\.113$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.147\.3\.40$ [OR]
RewriteCond %{REMOTE_ADDR} ^206\.105\.10\.198$ [OR]
RewriteCond %{REMOTE_ADDR} ^219\.163\.188\.218$ [OR]
RewriteCond %{REMOTE_ADDR} ^12\.148\.209\.196$ [OR]
RewriteCond %{REMOTE_ADDR} ^12\.148\.196\.141$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.102\.17\.41$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.102\.17\.65$ [OR]
RewriteCond %{REMOTE_ADDR} ^62\.36\.180\.86$ [OR]
RewriteCond %{REMOTE_ADDR} ^212\.27\.33\.116$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.103\.38$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.114\.112\.72$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.116\.207\.132$ [OR]
RewriteCond %{REMOTE_ADDR} ^68\.65\.210\.37$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.214\.112\.112$ [OR]
RewriteCond %{REMOTE_ADDR} ^216\.150\.216\.18$ [OR]
RewriteCond %{REMOTE_ADDR} ^82\.214\.194\.6$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.43\.240\.100$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.206\.21\.97$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.69\.120\.66$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.215\.87\.229$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.168\.0\.216$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.36\.5$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.157\.36\.4$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.215\.80\.51$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.215\.85\.192$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.215\.76\.148$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.252\.23\.186$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.161\.24\.131$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.161\.23\.245$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.81\.148\.131$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.145\.25\.105$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.15\.138\.214$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.94\.44\.127$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.128\.154\.91$ [OR]
RewriteCond %{REMOTE_ADDR} ^193\.252\.200\.122$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.11\.65\.210$ [OR]
RewriteCond %{REMOTE_ADDR} ^193\.251\.53\.22$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.11\.65\.74$ [OR]
RewriteCond %{REMOTE_ADDR} ^193\.251\.52\.59$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.49\.209\.131$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.14\.185\.210$ [OR]
RewriteCond %{REMOTE_ADDR} ^193\.251\.29\.45$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.13\.78\.25$ [OR]
RewriteCond %{REMOTE_ADDR} ^193\.253\.178\.183$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.51\.116\.43$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.13\.169\.152$ [OR]
RewriteCond %{REMOTE_ADDR} ^217\.128\.252\.166$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.48\.127\.67$ [OR]
RewriteCond %{REMOTE_ADDR} ^193\.251\.80\.30$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.248\.17\.52$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.13\.163\.248$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.150\.28\.220$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.51\.227\.244$ [OR]
RewriteCond %{REMOTE_ADDR} ^216\.119\.173\.253$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.76\.97\.228$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.6\.13\.215$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.6\.9\.217$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.86\.46\.195$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.72\.151\.218$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.174\.204\.122$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.174\.192\.77$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.101\.180\.170$ [OR]
RewriteCond %{REMOTE_ADDR} ^195\.144\.134\.42$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.14\.67\.140$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.14\.67\.42$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.15\.37\.100$ [OR]
RewriteCond %{REMOTE_ADDR} ^68\.109\.73\.115$ [OR]
RewriteCond %{REMOTE_ADDR} ^210\.200\.105\.227$ [OR]
RewriteCond %{REMOTE_ADDR} ^62\.195\.91\.228$ [OR]
RewriteCond %{REMOTE_ADDR} ^195\.166\.238\.131$ [OR]
RewriteCond %{REMOTE_ADDR} ^195\.166\.233\.191$ [OR]
RewriteCond %{REMOTE_ADDR} ^195\.166\.233\.65$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.226\.197\.211$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.93\.38\.130$ [OR]
RewriteCond %{REMOTE_ADDR} ^81\.199\.83\.31$ [OR]
RewriteCond %{REMOTE_ADDR} ^220\.144\.155\.229$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.88\.42\.40$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.31\.32\.16$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.37\.104\.196$ [OR]
RewriteCond %{REMOTE_ADDR} ^171\.75\.222\.63$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.88\.128\.12$ [OR]
RewriteCond %{REMOTE_ADDR} ^80\.218\.114\.94$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.21\.116\.139$ [OR]
RewriteCond %{REMOTE_ADDR} ^211\.21\.116\.130$ [OR]
RewriteCond %{REMOTE_ADDR} ^218\.160\.2\.47$ [OR]
RewriteCond %{REMOTE_ADDR} ^38\.112\.195\.5$ [OR]
RewriteCond %{REMOTE_ADDR} ^38\.112\.195\.9$ [OR]
RewriteCond %{REMOTE_ADDR} ^38\.112\.195\.51$ [OR]
RewriteCond %{REMOTE_ADDR} ^216\.185\.57\.98$ [OR]
RewriteCond %{REMOTE_ADDR} ^216\.138\.199\.77$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.61\.12\.100$ [OR]
RewriteCond %{REMOTE_ADDR} ^24\.123\.125\.205$ [OR]
RewriteCond %{REMOTE_ADDR} ^203\.67\.201\.21$ [OR]
RewriteCond %{REMOTE_ADDR} ^12\.219\.232\.74$ [OR]
RewriteCond %{REMOTE_ADDR} ^137\.15\.236\.98$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.124\.166\.234$ [OR]
RewriteCond %{REMOTE_ADDR} ^69\.156\.107\.85$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.60\.26\.248$ [OR]
RewriteCond %{REMOTE_ADDR} ^200\.106\.46\.26$ [OR]
RewriteCond %{REMOTE_ADDR} ^216\.71\.64\.19$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.94\.43\..*$ [OR]
RewriteCond %{REMOTE_ADDR} ^38\.144\.36\.13$ [OR]
RewriteCond %{REMOTE_ADDR} ^205\.217\.159\.2$ [OR]
#added 12/3/2004 BaiduSpider (email spammer)
RewriteCond %{REMOTE_ADDR} ^202\.108\.249\.185$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.108\.249\.177$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.108\.249\.182$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.108\.249\.184$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.108\.249\.189$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.135\.146\.200$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.135\.145\.221$ [OR]
RewriteCond %{REMOTE_ADDR} ^61\.135\.145\.207$ [OR]
RewriteCond %{REMOTE_ADDR} ^202\.108\.250\.196$
# BaiduSpider end

[wilderness]

Dan,
I'm not all that clear on whether your looking for IP ranges or something like this?

[jafsoft.com...]
[joseluis.pellicer.org...]
[botspot.com...]
[robotstxt.org...]
[psychedelix.com...]
[clearwaterbeachcam.com...]
[pgts.com.au...]
[projecthoneypot.org...]
[joseluis.pellicer.org...]
[iplists.com...]
[siteware.ch...]
[xoc.net...]
[spiderhunter.com...]

Don

[Intrpngn]

Since the identification can be almost anything, we review the logs and kill specific ipranges or addresses that consistantly misbehave.

The flaw with this is the fly by night spammer that harvests email addresses. To work around this we may be implementing forms and removing all email addresses from the sites.

If there is a better way to handle bots would certainly be game for it. This seems to work for us but not full proof.

also have no problems working with a group to put together a global list that is updated on a regular basis that can be called upon.

/later

[volatilegx]

Nice list of resources, Don. I guess what I am looking for is data relating only to web filters.

I don't think there is a resource that has a comprehensive list, which is one reason I want to create one.

I'm looking for both IP addresses and User-Agents. I want to associate the IP addresses with the User-Agents and also provide some background information about each filter.

Intrpngn, you have quite a list of IPs, but are they all content filters?

[wilderness]

Dan,
Please forgive me, I'm just attempting to clarify precisely what your looking for?
I just don't recall previously seeing the use of the term "filter"

Are you looking for something similar as to what is discussed and used extensively in the CLOSE TO PERFECT HTACCESS?
[webmasterworld.com...]

or are your looking for the crteria (visitor log lines) used to make the determination of either using a rewrite for an IP or UA?

Don

[volatilegx]

I am putting together a list of content filter spiders, by which I mean spiders like those for SurfControl, CYBERsitter, and other services which filter adult and other objectionable content from corporate networks, libraries, family PCs, etc.

I'm not sure all of these companies have actual spiders. Some probably rely on on-the-fly content screening. But I do believe many of the companies do keep databases of objectionable URLs and those are probably spidered.

I'd like to get their IP addresses, User-Agents, and behaviour patterns.

[wilderness]

[quote]I'd like to get their IP addresses, User-Agents, and behaviour patterns.[quote]

Dan,
I don't have my data organized in such a manner that would make it feasible or useful to another.
Even though at one time, I did begin a database, it was never completed. As a reuslt I use a rather flim-flam method of searching across my mail software and the previously exported mail files on my computer.

Offering the large accumulation of data that I have would be out of the question. The reason for these is that I tend to mix items of personal nature in with the storage areas of the data, which is related to the data. This storage method of directory structure is what assists me in locating my data.

I've had or recall seeing four contacts for joint ventures of organizing such things in the paste few days.

My primary interest is not in computers or websites, rather using the websites for the materials of which I have enough of a work load awaiting digitization to keep me busy until I'm in the ground.

My participation and activity here is related to an agreement I made to protect these copyrighted materials of which I digitize and make available on whatever it is I have;)
Whether your or I might choose to call my websites the internet or a modifed intranet?
There are more restrictions on my sites as related to visitor activity than might be imposed by most websites.

If you have specific inquiries concering a provider?
I may be able to assist you there. Part of my folder structure for the htaccess/filter thingy's is directory with many sub-directories of specific providers. Mixed in are also some bot directories and even some reigion directories.

Sorry!

Don

[jdMorgan]

The following is some code that I used to use. However, I have since decided to allow these content-filtering agents to access most of my sites, so as to prevent the sites from being blocked by the filters.

Content filtering user-agent IP addresses

# Websense
RewriteCond %{REMOTE_ADDR} ^63\.212\.171\.(12[89]¦1[3-9][0-9]¦2[0-4][0-9]¦25[0-4])$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.132\.15\.2(4[0-9]¦5[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.156\.198\.(6[89]¦7[4-8]¦8[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.194\.6\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.67\.32\.1(4[4-9]¦5[0-9])$ [OR]
#
# N2H2 Content Filtering
RewriteCond %{REMOTE_ADDR} ^66\.150\.1(6[89]¦7[01])\. [OR]
RewriteCond %{REMOTE_ADDR} ^206\.129\.[01]\. [OR]
RewriteCond %{REMOTE_ADDR} ^209\.102\.7[67]\. [OR]
#
# Netsweeper via Hamilton Hydro / FibreWired
RewriteCond %{REMOTE_ADDR} ^66\.207\.(9[6-9]¦1[01][0-9]¦12[0-7])\. [OR]
#
# Covenant Eyes
RewriteCond %{REMOTE_ADDR} ^69\.41\.14\.([1-9]?[0-9]¦1[01][0-9]¦12[0-7])$ [OR]
#
RewriteRule - [F]

Change all broken pipe "¦" characters to solid pipe characters before attempting to use this code.

None of these user-agents appears to be a spider. All are apparently triggered by the human visitor. Some are pass-through filters, and some are pass-by filters. That is, in some cases, the page requests come only from the IPs listed, and the response is fed back to the human user on the other side of the filter ("pass-through"), and in other cases, you see the human user IP address on the requests, shortly followed by the filter's requests from a different IP ("pass-by").

It's been awhile, and I did not record which were pass-through and which were pass-by. Also, I can't swear that none of them displayed any spidering behaviour, but if I recall correctly, they didn't.

It's a short list, but my real purpose in posting was to suggest that pass-through versus pass-by characteristics be tracked and noted. Other characteristics worth noting are whether the pass-by filters fetch only pages, or whether they also fetch images, CSS, external JS files, etc.

The pass-through type filters may be referred to as "filtering proxies" or "content firewalls" since these filters are typically located in a centralized, shared location -- either at the filter company's facility or at an ISP. Pass-through is the only way to absolutely guarantee that a page classified as objectionable can't get through the filter. This is what you need for "kid-safe" networks.

The pass-by filters are likely low-cost corporate "monitoring" systems, and probably do not check 100% of the URL in the requests passing by. Corporations do not require pass-through filters because they don't need to guarantee that nothing objectionable gets through. They just need a record so they can discipline/fire employees.

Jim

[wilderness]

Thanks for the clarification Jim.

These provides and insight into exactly what Dan is looking for.

Another is named SURF
In the third log line (I get these frequently) the IP is the actaul visitor rather than the software

194.203.75.29 - - [23/Jun/2003:13:20:42 -0700] "GET / HTTP/1.0" 200 9409 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"

213.173.146.3 - - [03/Sep/2003:04:25:19 -0700] "GET /myimage.jpg HTTP/1.1" 200 1162 "mysite" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; surfEU FI V3)"

68.43.81.97 - - [13/Sep/2003:14:40:12 -0700] "GET /myfolder/ HTTP/1.1" 200 5773 "-" "SURF"

[wilderness]

Google returned over 8-million hits on "content+filter+software (that link is at page bottom)

Here are some links from the first four pages.

Cybersitter
we-blocker
Net Nanny
GateFilter
CMS Praetor
Sentian & Bess and/or N2H2 [n2h2.com...]
Web Filter
SurfMonitor
Safe Eyes
SonicWALL
WebBlocker
PeaceFire
WebSense
BlockPorn
CyberPatrol
SafetySurf
Postfix
Enterprise Content Management (ECM)
Smart Filter

DMOZ list
[directory.google.com...]
Yahoo list
[dir.yahoo.com...]
More Info
[ks.securityfocus.com...]

Filter research
[tispa.org...]
Bypassing info
[security.nnov.ru...]

[google.com...]

[volatilegx]

Thanks, guys for putting me on the right track!