Forum Moderators: open

Message Too Old, No Replies

Can anyone give me a list of UAs I should always ban?

         

born2run

4:04 am on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hi so I need to get my invalid traffic mess sorted out sooner or later.

I'd like to know if there is any list of UA's that need to be blocked for sure? If there is any suggestion, kindly advise. Thanks!

tangor

4:54 am on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Many of the worst offenders have been exposed in this forum for years. keyplr routinely posts new bot alerts on a regular basis.

There is no "list" to speak of. Some bots are merely rude, others mean nothing (good or bad) and some are thoroughly bad. Start with the bots you want to let in and ban the rest (for now) and then allow bots in which can be of benefit.

Most folks consider b g y ddg to be good bots, for example.

keyplyr

5:11 am on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



@born2run - Never use a block list compiled by anyone, including any host.

Always do the research yourself to determine what is beneficial to your interests. There is no quick solution. You need to do the work. End of story :)

lucy24

5:56 am on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Can anyone give me a list of UAs I should always ban?
In a word: No.

born2run

9:20 am on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Ok yes I understand your points. I'll be doing my own research daily and try optimizing traffic to my site.

wilderness

4:30 pm on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I find it puzzling that this once active forum makes it so difficult for newcomers to learn! (rhetorical).
Is it so easy to grasp that we all began with 1+1=2 and not geometrical equations?

There are long time participants (fifteen years or more) that seem to add such heavy warnings of caution for that one good guy (bot) that offers an advantage, that the newcomer is faced with the impossible task of accepting defeat before they've even begun.

born2,
Here's some commonly misused UA's and are only meant for a guideline. You must explore each one with your own raw logs and determine for yourself whether using each word is detrimental or beneficial.
client crawl curl client download
Fetch grab Java Library libww
Nutch parser reaper Retrieve spider

These threads (Close to perfect htaccess) are more than fifteen years old. They are very long, repetitious and participants copy and pasted both UA's and syntax from unknown other sources, even if the syntax was incorrect (there are many, many errors).
With all the above in mind, the threads are still useful guidelines. Once again, you must explore each one with your own raw logs and determine for yourself whether using each word is detrimental or beneficial.

It is also imperative to note that after each and every modification (upload) of you htaccess that you confirm your site (s) are still working (not generating a 503).

[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]

As others have explained to you, there is no one-size-fits-all. Copy and pasting what one (or more) webmaster uses is not a sound practice, (at least not without exploring all the detriments to your site (s)). You need to analyze each addition and determine the overall benefit as related to your websites goals (only you know that).

lucy24

9:38 pm on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



after each and every modification (upload) of your htaccess ... confirm your site(s) are still working (not generating a 503).
Let's stress that, because some of us have learned it by sad experience :(

Immediately after uploading any changed htaccess file, request any random page on your site--even robots.txt will do if you're only concerned about 500-class errors--and make sure you don't get an error.

wilderness

9:49 pm on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



lucy,
I couldn't recall if there's a sig option at Webmaster World to include the above line and the detrimental/beneficial lines and was too lazy to go searching.

keyplyr

9:56 pm on Mar 7, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



We don't use sigs at WW.