Forum Moderators: open
We find a user agent (or bot?) called just U .
Would like to know what "U" refers to, to know if it's a Bot Or Not...
Thanks
It's likely just something somebody has done to circumvent the blank UA, however the only person able to confirm the logic, would be the person that created the UA.
I would suggest focusing on the IP and determining if the visitor has the capability to randomly change the UA "on the fly".
As you wade deeper into your logs you will start to find lots of what I call garbage user agents. Essentially, meaningless combinations of letters and numbers.
What I do is put them, along with their IP Address, into a database table, and when I see a new garbage user agent I query that database table for other IP Address matches.
It's not always helpful, but, for example, if you start to see multiple visits from the same IP Address using the same or different garbage UAs, you will probably want to take steps to block that IP Address from accessing your server(s).
Whether you ultimately decide to block any UA or IP Address is up to you. We can almost never tell you yes, you should absolutely block this thing. That's because what's garbage to us might be gold to you.
if you start to see multiple visits from the same IP Address using the same or different garbage UAs, you will probably want to take steps to block that IP Address from accessing your server(s).
Yes the problem is, say regular user gets FF and the "modify headers" addon then sets the UA and Referrer (and any other header) to whatever he likes. In many cases just to "promote" his own domains or whatever. And he's coming in from verizon, bell, comcast, aol, you name it. I don't see anything useful to block as all the ips are dynamic. Just resets the modem and gets another IP.
Now you cannot ban those IPs for sure. And there is nothing illegal about it. That kind of plugins are just for people who know nothing about browsers right?
Then of course you have servers who can expose some resource that points to your server hidden from view (like an image). So any client that connects to that server can pull stuff from yours too; unknown to him and unknown to you. Your logs in turn will show the domain of the spamy server who does that in the referrer, with the IP of the client. But again if you ban the client's IP you don't solve anything. And the other server never connects to yours so banning him is pointless.
This also helps spammers to counter hot-linking countermeasures if you think about it. Doesn't look good at all.
Instead of a 403 in that case you could send a redirect to a page explaining why the access was blocked. That maybe will be a bit more efficient than a plain 403 with an IP ban.
Now you cannot ban those IPs for sure.
If my servers are being abused by a range of IPs from Comcast, for example, I'll ban them in a heartbeat. I'll also redirect them to a form they can fill out to prove they're a human not a bot and then I'll whitelist them for the duration of that visit.
I'll ban entire countries if need be. On some of my servers I have huge countries like China banned.
That's part of what I meant about the decision to ban being one each webmaster must make. Just because I ban China doesn't mean you should as well. Just because you ban Russia doesn't mean I should too.
I've just as many North American IP's blacklisted due to actions of their customers as I do non-North American IP's (perhaps even more).
A "reliable" and sound business minded internet provider will make sure that localities are broken down and registered in sub-net ranges (although some may have network restrictions which prevent sub-net designations).
In many events, "we as webmasters" are required to be creative and initiate multiple conditions based on both IP and UA, as well as any other criteria we're able to muster.
Don
One of those pages has a mere four short lines and links to sub-pages and was intended to do exactly what your describing.
The page has existed for six years and is listed with the major SE's. It does return on SERPS for the key-words related to the directory/category and the sub-pages.
"Rank" is another issue, even though in this instance it's not a concern for me.
