homepage Welcome to WebmasterWorld Guest from 54.205.7.136
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Funnelback
Pfui




msg:4375620
 9:49 pm on Oct 17, 2011 (gmt 0)

Australia-based search company owned by Australia-based Squiz, a CMS company. [en.wikipedia.org...]

122.99.95.245.static.transact.net.au
Mozilla/5.0 (compatible; Funnelback)

robots.txt? Yes

IP has PHP history for bot-running the same UA. [projecthoneypot.org...]

Odd that the company opted to omit bot-UA format basics like version numbering, an info page, even a site URL.

 

dstiles




msg:4376191
 8:46 pm on Oct 18, 2011 (gmt 0)

Suspicious IP block. Until today I took DNS's word that it was for mobile DSL but the range includes statics and web sites - not a mobile activity as far as I'm concerned. 122.99.64/19 now banned.

Mokita




msg:4376224
 9:47 pm on Oct 18, 2011 (gmt 0)

TransACT provide broadband and phone services in and around the ACT, Australia. Mostly to residential customers:

[transact.com.au...]

Whilst a the odd occasional single IP might be used for nefarious purposes (as can happen in any reputable company offering hosting), the vast majority is benign.

I also consider Funnelback to be benign, and if I felt it offered no benefit to my websites, I'd ban it via UA, not IP. Especially not a range that could lock out a swag of potential visitors.

YMMV.

Pfui




msg:4376256
 11:24 pm on Oct 18, 2011 (gmt 0)

Ditto. Deny the UA at this point; and/or, given its history, that single IP (not even the Class C).

Mokita




msg:4376328
 2:18 am on Oct 19, 2011 (gmt 0)

I haven't seen Funnelback recently - only about 5 years ago:

[webmasterworld.com...]

If I remember correctly, it wasn't a pest. So I didn't bother much about it.

Interesting that back then, the UA did include a site URL/robot info page.

I probably should also clarify "the ACT" from my earlier post.

The Australian Capital Territory (A.C.T.) contains Canberra, the nation's capital. A little bit like Washington DC, except it also includes large areas of national parks, and farms.

dstiles




msg:4376789
 9:39 pm on Oct 19, 2011 (gmt 0)

I said "suspicious" because an IP range advertising in DNS as dynamic - and mobile at that - should not, in my opinion, include web sites. I'm not even sure how you could host a web site on a dynamic mobile IP. Hence my assertion that there is something not quite right about the range.

Mokita




msg:4376801
 10:03 pm on Oct 19, 2011 (gmt 0)

Please would you let me know where you are seeing the DNS advertised as mobile/dynamic?

dstiles




msg:4377293
 8:29 pm on Oct 20, 2011 (gmt 0)

Sorry, I thought I'd seen it in DNS as that but DNS actually says "IP Service Delivery Network", which is somewhat ambiguous (as are many DNS entries). Not sure where I got the "mobile" bit from now - possibly from their web site as "phone" provider.

I had originally listed the range in my database as dynamic until this thread, when a bit more investigation showed me the hosting etc aspect on the higher IPs. I accept that SOME of the range may be dynamic/static DSL but the higher IPs are suspicious (to me) so I invoke my right to block them. :)

Regarding the UA that pfui posted: that is certainly some kind of bot UA (or a badly mangled browser or proxy that had to have been fiddled with).

Mokita




msg:4377561
 9:16 am on Oct 21, 2011 (gmt 0)

@dstiles
Many thanks for your response.

I have to admit I was quite puzzled - as TransACT's raison d'etre is that they were the very first cable company in the ACT. Their telephone services are an after-thought/commercial add-on, getting customers to bundle their services. So heaps more of their IPs are cable/static than mobile/dynamic.

... so I invoke my right to block them. happy!


Fully support your right to block anything - even Google if you so wish! I'm sure all the regulars of this forum adhere to the principle of "Do what's right for you/your website" - me included.

I just wondered if you realised that you would also be blocking a very large number of domestic and totally benign business consumers.

Regarding the UA that pfui posted: that is certainly some kind of bot UA (or a badly mangled browser or proxy that had to have been fiddled with).


No argument that it is a bot, that has been totally evident since the day it first appeared. But in my experience it is both benign and very rarely seen - therefore it is not a pest nor any danger to our websites, that I can discern. Also, I am not so extreme as to ban whole IP ranges for the sake of one bot that could be controlled otherwise (if I so desired). I only do that if the range is patently all host/colo etc., or located in a highly suspect country like Ukraine, Russia etc.

I note that it requests robots.txt, and I haven't seen anything yet that indicates it violates/disregards it - so that would seem to indicate it is polite. Only if I saw evidence that it was disregarding robots.txt, would I invoke stronger measures.

So, I reiterate what I said previously:
... if I felt it offered no benefit to my websites, I'd ban it via UA, not IP. Especially not a range that could lock out a swag of potential visitors.


All above said, purely to provide both background and my personal opinion. No criticism whatsoever implied if you disagree. ;)

dstiles




msg:4377798
 7:32 pm on Oct 21, 2011 (gmt 0)

Thanks for the thoughts but the actual range I'm blocking is only 8192 IPs (122.99.64/19). Those I would deem "dodgy" are actually about 512 IPs - the top /23.

But a compromise: I'm now blocking 94/23 and allowing the rest of the /19. :)

Have to say I'm also a bit more careless about certain ranges such as RU, UA, CN, KR, VN, BR than others when it comes to blocking. I do a rough check: if it looks server-ish it's blocked, otherwise it's put on probation (I have a category called "badcountry" which a small proportion of my UK-centric sites reject). I sometimes think wistfully of adding US to the list... :)

Mind you, I'd also like to block UK ones at times, and I live here! :(

Mokita




msg:4378238
 6:47 am on Oct 23, 2011 (gmt 0)

Mind you, I'd also like to block UK ones at times, and I live here!


Me too! I see broadband/consumer IPs, from top ISPs in my country, exhibiting highly iffy behaviour. But I don't dare ban them, in case I lock out real people. <huge sigh>

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved