Forum Moderators: open
Grabbed seven pages.
No robots (of which this robot is specified.)
No images.
69.38.159.128 - - [29/Dec/2004:23:52:09 -0800] "GET /myfolder/mypage.html
HTTP/1.1" 200 21438 "-" "FAST Data Search Document Retriever/4.0 04091300 2004-09-13 15:40:33 UTC"
Denied the entire range.
There were two visists from other ranges of the provider to the TOS page I quoted in my email.
One of those secondary visits was from one of the providers sub companies which sells firewall hardware.
Those two scondary ranges have been added to my denies as well.
Do be careful banning traffic from "firewall" companies. Many times, these accesses are from *users* of those firewalls, and the firewall is actually fdoing the page fetching. If you block appliances from companies like SonicWall, etc., you are actually blocking the user behind the firewall, and that user may not even know that he or she is behind a firewall. He/she will think your site is broken.
Not that I don't feed out quite a few 403s myself, just be careful about 'em... ;)
Jim
Many times, these accesses are from *users* of those firewalls
In the event that an equestrian and breed specific visitor (as related to the content of my sites)has a compelling need to enter beyond a firewall which presents a UA previously defined (or anything similar such as XXXX and more?)
Than that equestrian visitor has no compelling need or reason to view the content of my pages. Those pages are rather simple under the theory of KISS.
My sites do not utilize cookies and only about one-third of the pages utilize a java script for a site counter (and even that will be eliminated eventually.)
I do not utilize ActiveX or Flash or ASP or PHP or Perl or anything. Instead I serve straight text with accompanying images.
Most of the visitors to my sites may gain a valid identification of myself within the industry of which they participate and through regulatory bodies within that industry.
Their need to present a firewall (or blocking of data and use) not only restricts the possible enhancement of my websites, but restricts the continuity of the industry they participate in.
How?
The blocking of what precisely they are searching for reduces my chances of providing new content based on what they are searching for. In fact, if a visitor frequently (and before I initated a form of contact which you were kind enough to assist with [thanks again],) returns to my site looking for something which is obvious to me however not to the visitor, I improvise a page by adding specific content for their answer.
In most instances, I provide digitized content of which they cannot find any place on the inernet or even within their own industry.
The least that visitor may do in return for my extensive efforts is visit my sites with a "some what method" of allowing me to separate them from all the robots and other harvesters ;)
Don
I'm not real happy that some of these devices --mostly used by ISPs and corporations-- hide user-agents, referrers, and other information that's useful to us as webmasters. But I'm advocating in favor of the "clueless user" here, and saying that blocking their access due to factors beyond their control is unfair and possibly unwise.
The cheapy personal firewalls like Norton Internet Security and ZoneAlarm are more likely to indicate one user who has one of these programs installed. It's more likely that they do know it's there -- although there are many folks who buy new PC's with Norton pre-installed and haven't got a clue how to configure it or turn it off. But accesses from something like SonicWall indicates that someone has spent maybe $850 on one of these units (plus more for anti-virus and anti-worm filtering subscriptions), and that's rarely a single-user application. It's much more likely to be a corporate firewall, or one of the increasingly-popular "safe ISP" services which use services like WebSense to classify and filter content on-the-fly.
The people who sign up for Safe ISP service, and the corporations who install firewalls between their LAN and WAN are battling the same enemy we are -- the people who feel entitled to abuse the resources of the internet. So, some balance and perspective is needed when considering these issues. I fully support the right of any Webmaster to block any accesses he/she doesn't want. But it can be very important to the success of your site to do it in a fully-informed way... It's critical to understand *what* your are blocking. We can only hope that firewall makers and Safe ISP service providers will catch on and help Webmasters make more-informed decisions by providing appropriate information in the HTTP request header.
Jim
