1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 4:24 pm May 4, 2026

Forum Moderators: open

Message Too Old, No Replies

Is this a human?

         

aristotle

4:31 pm on Jun 2, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I saw this in my Latest Visitor logs for one site today. The 403 responses are due to the word "Desire" in the UA. I don't remember what caused me to block UAs containing this word, but now it looks like I might have blocked a real human in this case.
Host: 205.197.242.159
/
Http Code: 403 Date: Jun 02 12:14:35 Http Version: HTTP/1.1 Size in Bytes: 13
Referer: -
Agent: Mozilla/5.0 (Linux; Android 4.4.2; HTC Desire 510 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36 ACHEETAHI/2100501044

/favicon.ico
Http Code: 403 Date: Jun 02 12:14:35 Http Version: HTTP/1.1 Size in Bytes: 13
Referer: -
Agent: Mozilla/5.0 (Linux; Android 4.4.2; HTC Desire 510 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36

The IP lookup shows:
IP: 205.197.242.159
ISP: XO Communications
Organization: Jasper Technologies
Services: None detected
Type: Wireless Broadband
Assignment: Static IP
Country: United States

It annoys me when I discover that I blocked a real human. Should I remove the word "desire" from my UA block list?

keyplyr

9:43 pm on Jun 2, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Looks like a mobile user to me. The UA is common as is the ISP XO Communications. The HTC Desire is a popular phone in Europe as well as the USA. I had one for a while.

aristotle

12:05 am on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks keyplyr
This happens occasionally, where something I do in .htaccess later blocks a real person and I have to take it out.

lucy24

12:13 am on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



It annoys me when I discover that I blocked a real human.

Hear, hear. It's why I have exemptions for favicon (and apple-touch-icon) and stylesheets. Sure, logs will show that a human asked for them, but I'd rather let them see what the 403 page is really supposed to look like.

Requests for the favicon will normally come in with no referer, unless your HTML explicitly links to a favicon. This makes them an especially good clue that there was a human involved; even infected browsers that follow redirects and ask for stylesheets rarely put in a favicon request.

aristotle

12:25 am on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks Lucy -- I remember you mentioning in another thread somewhere that a request for a favicon is nearly always associated with a real person.

But I prefer not to have a 403 page because I think it makes the site more secure.

Pfui

1:49 am on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



But I prefer not to have a 403 page because I think it makes the site more secure.

Hmm. Well, I'm not sure how a plain, top level /403.html would give bad actors a way in as-is, but here's a workaround you might consider if you're an Apache person:

I put my very simple 403 page html code "in" .htaccess, as part of the ErrorDocument 403 directive. The whole thing, from <HTML> to </HTML>.

You can also just use plain English. [httpd.apache.org...]

keyplyr

2:26 am on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



But I prefer not to have a 403 page because I think it makes the site more secure.
Just never include any incoming links to your account from the custom 403 page. I've seen people do it and it amazes me to their thinking... "I'll block you from access to the page you requested because you're a threat, but here is another page to hack into my account."

Most hosting companies serve their own 403 page (in fact it is the default with most servers configs) so you really don't need to design a custom page, it's just that most people like to in case visitors who are lost can get a more appropriate greeting than FORBIDDEN ! :)

lucy24

4:32 am on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I don't care if they know what the pages are called; they're still not allowed to get there, and that's all that matters. Especially if the page in question outweighs the 403 page by a hundred to one.

Besides, they already know the page titles. Some humanoid fifth-columnist friend of theirs stopped by the week before last and made inquiries.

My host has a default error-document name, but the document itself doesn't exist unless you create it on your own site. So on brand-new sites you can look in error logs and find that each 403 is followed by a failure to find the document "forbidden.html" in the domain root.

keyplyr

9:13 am on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Some humanoid fifth-columnist friend of theirs stopped by the week before last and made inquiries.

So that's who that was?

aristotle

1:41 pm on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



keyplyr wrote:
Most hosting companies serve their own 403 page (in fact it is the default with most servers configs)

I'm not sure that's the case with the two hosting companies I use. If you look at my first post in this thread, you'll see that no 403 page was returned, but only a header with a 13-byte message. I have 5 sites on three different servers at two different hosting companies and all of them respond the same way.

But I don't have time to say anymore right now -- I'll have to come back later to finish

aristotle

4:13 pm on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Actually what I have in my .htaccess is
ErrorDocument 403 "Access Denied"

Anyway, the main reason I do it this way is help reduce the server load in case of a rapid succession of invalid requests. Last year one member of this forum, in a discussion about hypothetical situations, described what I would call a half-hearted DDOS attack from a small botnet. In such a case, the server has a better chance to keep up if it doesn't have to serve real 403 pages of several hundred bytes.

Several of my sites get numerous daily requests from what appear to be botnets, in a kind of "slow-motion attack", but these could conceivably be preludes to an eventual all-out attack. At least one of these botnets, if that's what they are, appears to be petering out, so it might not be able to stage a strong attack.

Yes I realize that this is mostly speculation about marginal scenarios. I also know that some members here try to use a 403 page to give real humans a backdoor way into the site if they are accidentally denied on their first attempt. But that's not how I like to do things -- I'd rather put my efforts into avoiding blocking them in the first place.

lucy24

5:39 pm on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



the server has a better chance to keep up if it doesn't have to serve real 403 pages of several hundred bytes

Well, that's a question of fact, which may or may not have an answer. At bottom the server does two things: evaluate requests (internal activity) and send out a response (external activity). Is the actual size of the response (headers plus content) a meaningful part of the server's workload? It's possible the answer is no, considering that some hosts no longer even measure bandwidth, but only CPU/RAM usage. That's assuming for the sake of discussion that they charge money for whatever costs them the most money to provide.

Now, I do have additional access-control rules for a half-dozen or so exceptionally large pages. But that's for cases where the HTML alone can be a megabyte or more.

aristotle

6:12 pm on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Is the actual size of the response (headers plus content) a meaningful part of the server's workload? It's possible the answer is no

Yes you could be right about this. Even if you are, I still don't see any drawbacks to the way I'm doing it. Unless I wanted to either have a "pretty" page to tell someone that they're not wanted. Or as I mentioned, to provide a "backdoor" into the site for real humans. I don't think either of these options is worth the possible risk of slowing down the server, in case it does.

tangor

9:13 pm on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I like to keep my .htaccess as black and white as possible. It is when you play in the gray areas you get in trouble. I use the server 403 as it is short, sweet and concise.... and if I messed up the .htaccess with a filter term then it is up to me to correct that error. A 403 is special. It is intended. I am the one who coded it and... at the time... I meant it! And in that regard I don't provide any additional info, or links on the 403.... that's what my 404 is for (sloppy typing among other reasons), but a deny is a deny and that's my black and white.

Is this a human? ... Could be. Is "Desire" a word I'd filter? No... not without a modifier like "hot" or some other term. So... in this case, I'd take the filter out and wait and see what happens.

trintragula

3:24 pm on Jun 4, 2015 (gmt 0)

10+ Year Member Top Contributors Of The Month



On my site the "403" is a register/login prompt, which is about 20K, not including the nice pictures, and includes a few links. On an average day I'll send out about one of these per minute, almost all of which will be received by robots. And once in a long while, a human uses it and in so doing clears their access again.
I've never seen it used for DDOS.

lucy24

4:15 pm on Jun 4, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



the "403" is a register/login prompt

That makes it sound as if you're using a 403 as a 401. Would malign Ukrainian robots be allowed into your site if they presented suitable login credentials?

keyplyr

7:07 pm on Jun 4, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I like to keep my custom 403 as small as possible. It gets got about 3k times per day. Nothing but text, two packets sent, one http connection:

Forbidden
___________________

Permission to access this server is denied.

Possible Reasons:
• You are in violation of copyright.
• You are hiding your browser or user agent.
• You are using a tool or method not allowed.
• Your host/ISP has been banned for bad behavior.

Your IP Address: ##.####.## has been logged.

trintragula

8:13 pm on Jun 4, 2015 (gmt 0)

10+ Year Member Top Contributors Of The Month



Would malign Ukrainian robots be allowed into your site if they presented suitable login credentials?

Yes they would, but to do that they'd have to get past the spam blocker, which is probably impossible without human help.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 4:24 pm May 4, 2026