One of Oldies - my Favorites - Crawler, Spider, and User Agent ID forum at WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

One of Oldies - my Favorites

W3C//DTD HTML 4.01 Transitional//EN

blend27

3:25 pm on Jan 16, 2023 (gmt 0)

Some Time Ago!

[webmasterworld.com...]

time.....

not2easy

4:46 pm on Jan 16, 2023 (gmt 0)

Indeed. It seems older than it is. Much has changed.

blend27

5:30 pm on Jan 16, 2023 (gmt 0)

Yep, I was a brunet back then, full speed... not2easy, my profile joined date is 2 years older, to the day...grrrr. :)

lucy24

6:01 pm on Jan 16, 2023 (gmt 0)

This prompted me to do a global search for the string �HTML 4.01� throughout the �www pages� superdirectory. Result: 52 hits, most of them in directories whose name include � OLD� meaning that they have long since been superseded. Also a few rather elderly local files, often with names including "2012" or "2013", so let's just update those in case I ever have occasion to revisit those particular pages.

The very last hit, amusingly, was something on my test site called pageold.html that seems to have been created for the purpose of testing a redirect (ask for pageold.html, get sent to pagenew.html), so the page headed html 4.01 would never actually be seen. But I updated it anyway.

tangor

11:10 pm on Jan 16, 2023 (gmt 0)

The real fun about old memories is knowing that IE is not really an issue any longer. :)

lucy24

6:37 pm on Jan 17, 2023 (gmt 0)

:: quick run to current logs ::

6596 occurrences of MSIE, of which:
� 6279 403
� 196 418 (my host�s response to mod_security, generally thanks to an ancient UA string involving MSIE 6)
� three 429 (response that started cropping up in logs in October)
� 22 302 (used for some types of disallow)
... leaving 84 that got in, either at once or via 301.

:: much business with calculator before figuring out that I came up short because two of those 6596 said MSIE twice, so it's really only 6594 ::

As far as I can make out, not one of those 84 was a legitimate human, with the possible exception of someone using MSIE 7 on an elderly tablet, and another someone coming in from a bad neighborhood--and, well them's the breaks.

Yup. I can simplify access controls by simply blocking MSIE regardless of number. Yay.

blend27

9:49 pm on Jan 17, 2023 (gmt 0)

OT

--someone using MSIE 7 on an elderly tablet--

No Such thing, I am not that Old ;)

lucy24

10:20 pm on Jan 17, 2023 (gmt 0)

Well, I've never professed to know anything about MS tablets. And if they were lying about their UA, they definitely deserved to be locked out :) My previous cutoff was MSIE 8 and below; this particular visitor appeared to be human because they got the various supporting files associated with a 403.

blend27

4:07 pm on Jan 18, 2023 (gmt 0)

---because they got the various supporting files---

----------------------------------
403.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<--! ^^ :) ^^ -->
<head>
<link href="/funny/my403.css" rel="stylesheet">
 </head>
 <body>
<h1></h1>
<div class="403">403!</div>
<div id="message">
</div>
<script src="/funny/my403.js"></script>
</body>
</html>

.htaccess

#----------------------------------
RewriteRule ^funny/my403.js real-funny/myReal403.js [QSA, L]
RewriteRule ^funny/my403.css real-funny/myReal403.css [QSA, L]
RewriteRule ^funny/403-background.jpg real-funny/403-realBackground.jpg [QSA, L]
#----------------------------------

IMPORTANT:
Create a directory called real-funny and place 3 files(myReal403.js, myReal403.css and 403-realBackground.jpg), coded below in it.
-----------------------------------

myReal403.css

div.403{
background-image: url("/funny/403-background.jpg");
background-repeat: repeat;
}

myReal403.js

//----------------------------------
var mW = 0, mH = 0;
if( typeof( window.innerWidth ) == 'number' ) {
//Non-IE
 mW = window.innerWidth;
 mH = window.innerHeight;
 }
 else if( document.documentElement && ( document.documentElement.clientWidth || document.documentElement.clientHeight ) ) {
 //IE 6+ in 'standards compliant mode'
 mW = document.documentElement.clientWidth;
 mH = document.documentElement.clientHeight;
}
else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
 //IE 4 compatible
 mW = document.body.clientWidth;
 mH = document.body.clientHeight;
}
var OldDiv = document.getElementById('message');
var newDiv = '<div align="center"><img " src="/funny/403-background.jpg?u='+ mW + '.'+ mH +'"></div>';
OldDiv.innerHTML(newDiv);
//----------------------------------

What this will do, to your webserver logs:

1. 403.html will render based upon you 403 rules. << 1 HIT to HTML
2. div.403 will have a background(as per CSS if requested) << so at this point that you will know that CSS file was actually requested and browser renders CSS by requesting background-image for that div. << 2 Hits = one for CSS and one for the image.
3. if js file requested(+ one hit), and IF/When JS executed a new entry will be created with an image(+ one hit in the logs), the path to the image in the logs will be /funny/403-background.jpg?u=1024.800 (gets inner width of the browser, diff for every browser)

At this point you have validated that you have a 403 file requested, browser is capable of requesting & rendering basic CSS, requesting and rendering/executing basic Javascript.

Total 5 hits per 403 and that is why my Full Head of Hair is almost totally GREY ;)

Tip: for .htaccess rules and file locations, real-funny directory could be outside of the website's programmatic rules, so less overhead and if you are using any type server-side scripting language, the Sky is the Limit. You could add the same basic logic to your website if you want to, I don't know, track users?....

For Example: Since of at least version 80, Chrome(& based) will pass sec-fetch headers, so will Firefox and Edge << if files are requested via HTTPS << at is least 3 rules right there.

....and time for a haircut appointment-visit for me.

_{Disclaimer: there might be syntactical errors in the code above so test it on you local server before.}

lucy24

7:07 pm on Jan 18, 2023 (gmt 0)

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

.... which is where we came in, heh heh.

But I don't see why the rewriting is necessary; isn't it just more work for the server?

:: detour to logs to refresh memory ::

Counting the initial blocked request, there will be four or five external requests--depending on whether they ask for favicon, never explicitly referenced in HTML--in main site's logs, and possibly a piwik request on my personal site, where all analytics files live. Between the first two external requests come the internal requests: first of course for the 403 page itself, and then its shared includes, notably logheaders.php so that I can, if I so choose, find out why the request was blocked.

:: further quick run to logs, followed by business with calculator ::

Overall, less than 1% of blocked requests are followed by a request for errorstyles.css, meaning that less than 1% of 403s are either human or trying hard to pretend. And only about 1/5 of that 1% request piwik.php. When I run logs I flag requests for errorstyles.css and piwik-for-403-page ... but then 99 times out of 100 I don't bother to investigate further.

In general I prefer not to block humans, unless they're from {country} or {other country}.

blend27

8:26 pm on Jan 18, 2023 (gmt 0)

--- But I don't see why the rewriting is necessary---

1. Bot requests HTML << Pass
2. Bot Parces HTML and requests attached file << Pass
3. Dump Bot does not Parse CSS and nor requests images mention in CSS file << No pass
4. Dump Bot does not Parse JS and nor requests images mention in JS file<< No pass

Basic browser will get a score of 4, so will the SMART Bot at lease 3 on a second run(Google Bot or Bing do not score 4, they do not request '/funny/403-background.jpg?u='+ mW + '.'+ mH').

SUPER SMART BOT WILL request all files, Google has one with vertical(innerHeight of 20000, i am not kidding). So will Modern Browser.

-funnyRequests, from SUPER SMART BOTS that have 1600x1200 width and height of the browser, no such thing as far as I know, especially for Samsung Galaxy S7(SM-G930V).


205.169.39.13-----1600x1200 -----Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36
205.169.39.19-----1600x1200 -----Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36
205.169.39.36-----1600x1200 -----Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36
205.169.39.55-----1600x1200 -----Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36
65.154.226.165-----1600x1200 -----Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36
65.154.226.166-----1600x1200 -----Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36
65.154.226.167-----1600x1200 -----Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36
65.154.226.169-----1600x1200 -----Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36
65.154.226.170-----1600x1200 -----Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36

205.169.39/24 ----- CenturyLink Communications, LLC << big time problem makers
65.154.226/24 ----- CenturyLink Communications, LLC << big time problem makers

----- isn't it just more work for the server? --

This is where the fun begins, In my case because I like to log everything in DB and build reports based on that:

RewriteRule ^funny/my403.js real-funny/myReal403.js [QSA, L]

is actualy:

RewriteRule ^funny/my403.js [b]ColdFusionTemplate.cfm[/b] [QSA, L]

Could be any resource type, really, as long as server sends back PROPER header for MIME type requested. And I could at the same type make a webservice request to block that IP on other sites,

Rrrrrrealy there is no end to it.....

I don't even look at the server logs at this point, it is all automated unless there is discrepancy in a rule that I am not aware of, everyone goes crazy their own way :)...

---- unless they're from {country} or {other country}---

Don't we all...

tangor

1:52 am on Jan 21, 2023 (gmt 0)

This CSS thread sure sounds a lot like a SPIDER thread! (And might be very useful there as well!)

blend27

12:01 pm on Jan 21, 2023 (gmt 0)

I would totally not mind if a portion of this be sliced-off into a new one over @SPIDERS

blend27

3:54 pm on Jan 21, 2023 (gmt 0)

@last! where ^^ supposed to be, I did hijack my own thread thought.

Thank You @not2easy.

lucy24

5:36 pm on Jan 21, 2023 (gmt 0)

Although it was in the CSS subforum, it was never a CSS thread. It started out about HTML and then, hm, began to wander...

Come to think of it, SSID isn�t an entirely accurate subforum title either, since much of it is about dealing with unwanted visitors. It makes me think about how sometimes the most detailed information about the habits of certain insects or small mammals can be found on ... drumroll ... exterminators' sites.

not2easy

6:13 pm on Jan 21, 2023 (gmt 0)

I think it might have begun with a different purpose, but this is where the habits of bots vs. humans is most commonly found ;)