Forum Moderators: phranque
Sec-Purpose: prefetch;anonymous-client-ip
Purpose: prefetch
I block requests with “Purpose: prefetch”. (This may be redundant; a quick riffle through logged headers shows no other possible value of the “Purpose” header, so its bare existence should be enough.) I only find it from Chrome, but that does take care of the Androids, which seem to be your main trouble. pratically all these webpage requests come from a unique IPDo you mean that each one is different? If so, it's probably not worth blocking by IP. But you might look up a few and see if they're coming from servers/colos or from human ISPs.
When I check my Apache log files, I can see hundreds of connections (GET) per minute, each from different IP (not bots, all coming from Ireland, New Zealand, Australia, Canada, UK, and USA).
For the last week or so I have been hit with an inordinate number of robots--say, ten times as many as normal--all claiming to be Androids using relatively elderly Chrome (mostly 40s and 50s).
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2691.1542 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.4061.1315 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.4061.1315 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2425.1227 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.9131.1527 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.4578.1932 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.4578.1932 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.4578.1932 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.4578.1932 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.6026.1344 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.7471.1660 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.6830.1011 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.4015.1616 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.1045.1416 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2444.1604 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.8781.1673 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.8638.1067 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.6077.1923 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.1371.1249 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2856.1017 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2744.1720 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.2787.1209 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.5199.1026 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.4702.1426 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7652.1913 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7848.1375 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.9605.1264 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.8505.1911 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.3306.1078 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.5294.1891 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.6871.1827 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.6871.1827 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.7095.1139 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.8195.1756 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2247.1336 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2247.1336 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.6637.1618 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.1923.1904 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.5499.1055 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.1214.1806 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.4776.1127 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.4776.1127 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.6137.1116 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.6137.1116 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.8911.1677 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2412.1508 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.1004.1784 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2300.1888 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.5972.1720 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.6026.1344 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.6026.1344 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.3797.1105 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.3397.1381 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.3646.1456 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.3646.1456 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.4651.1390 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.8721.1968 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2800.1113 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.8193.1247 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2878.1428 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3272.1436 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.5102.1629 Mobile Safari/537.36"
"Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.6683.1747 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.3835.1676 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.3835.1676 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.8942.1265 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.4945.1962 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2635.1109 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.8406.1640 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.4555.1489 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.8164.1752 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.4832.1349 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.8862.1240 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.9731.1481 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.7100.1154 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.9715.1724 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.9715.1724 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.9855.1093 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.8613.1157 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.2034.1419 Mobile Safari/537.36"
"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3956.1284 Mobile Safari/537.36"
[edited by: gronetwork at 1:32 pm (utc) on Dec 19, 2023]
What is the best method to log all the headers ?Someone else has asked that before and lucy24 shared basic info here: [webmasterworld.com...] though there may be updates since that 2018 thread. She's far better at this than I am.
BrowserMatch Chrome/[56]\d\. old_chrome=$0
BrowserMatch Android !old_chrome
BrowserMatch Chrome/[1-4]?\d\. old_chrome=$0
(leading to a Require env old_chrome among many others) meaning that Chrome < 50 is unconditionally blocked, while the 50s and 60s get a pass if it's Android. I update this kind of thing every year or so by checking if there have been requests for .css or /piwik/ (the directory still has this name although it's now Matomo); if not, it can be consigned to robot-dom. there may be updates since that 2018 threadHere's what mine currently looks like. It grew out of something incrediBill posted a few years earlier, so there are parts I personally don't understand. I added output buffering a couple years ago in hopes it would prevent tangles when two requests came too close together. (It didn't; I just haven't got around to removing that part.)
<?php
// shared header function
function get_server($var)
{
return isset($_SERVER[$var]) ? $_SERVER[$var] : false;
}
if (!function_exists('getallheaders'))
{
function getallheaders()
{
$headers = '';
foreach ($_SERVER as $name => $value)
{
if (substr($name, 0, 5) == 'HTTP_')
{ $headers[str_replace(' ', '-', ucwords(strtolower(str_replace('_', ' ', substr($name, 5)))))] = $value; }
}
return $headers;
}
}
if (!function_exists('getenvif'))
{
function getenvif()
{
$envvar = getenv();
return $envvar;
}
}
ob_start();
$ip = get_server('REMOTE_ADDR');
$fh = fopen($_SERVER['DOCUMENT_ROOT'] . "/boilerplate/headers-". date('Ymd') . ".log","a");
fwrite($fh, date('Y-m-d:') . date("H:i:s\n"));
$thispage = $_SERVER['REQUEST_URI'];
fwrite($fh, "URL: $thispage\n");
$status = $_SERVER['REDIRECT_STATUS'];
fwrite($fh, "Status: $status\n");
$secure = $_SERVER['HTTPS'];
fwrite($fh, "HTTPS: $secure\n");
fwrite($fh, "IP: $ip\n");
fwrite($fh, "----\n");
foreach (getallheaders() as $name => $value)
{
fwrite($fh, "$name: $value\n");
}
fwrite($fh, "----\n");
foreach (getenvif() as $name => $value)
{
// if (preg_match ('/(REDIRECT_)?[a-z]/',$name) && $value)
// exclude gzip_only_text_html
if (preg_match ('/^[abce-z]/',$name) && $value)
{
fwrite($fh, "$name: $value\n");
}
}
fwrite($fh, "----\n\n");
fclose($fh);
ob_end_flush();
?>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "Pixel 2 Build/OPD3\.170816\.012" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Nexus 5 Build/MRA58N" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "iPhone OS 11_0 like Mac OS X" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "SM-G900P Build/LRX21T" [NC]
RewriteRule ^ - [F,L] <?php
$useragent="";
if(isset($_SERVER['HTTP_USER_AGENT']))$useragent=$_SERVER['HTTP_USER_AGENT'];
//echo "<br>Current user agent : ".$useragent . "<br>";
if (strstr($useragent, 'Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012')
|| strstr($useragent, 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N')
|| strstr($useragent, 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X')
|| strstr($useragent, 'Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T')
) {
die('System error!');
}
else{
//echo '<br>You are OK.';
}
?> (Capture|Client|Copy|crawl|curl) escape spaces, slashes and punctuationIn mod_setenvif, happily you can deal with spaces by putting the whole thing in quotation marks. In mod_rewrite, you have to escape them. (And beware of trailing spaces, or your server will explode!)
divided with a vertical line likeTechnically known as a pipe ;) If you're checking for more than one possibility of the same thing--user-agent, IP, referer, filename--you can collapse them all into a single line, as long as you don't let the line get too long. This applies both to Conditions and to the Rule itself, as in the typical hole-poking rule along the lines of
RewriteRule ^(forbidden|missing|gone|repairs) - [L]
<?php
$useragent="";
if(isset($_SERVER['HTTP_USER_AGENT']))$useragent=$_SERVER['HTTP_USER_AGENT'];
//echo "<br>Current user agent : ".$useragent . "<br>";
if (strstr($useragent, 'Pixel 2 Build/OPD3.170816.012')
|| strstr($useragent, 'Nexus 5 Build/MRA58N')
|| strstr($useragent, 'iPhone OS 11_0 like Mac OS X')
|| strstr($useragent, 'SM-G900P Build/LRX21T')
) {
http_response_code(403);
die('Forbidden');
}
else{
//echo '<br>You are OK.';
}
?>
if(isset($_SERVER['HTTP_USER_AGENT']))
{ do stuff }
else{
//echo '<br>You are OK.';
}I would think that if there is no ($_SERVER['HTTP_USER_AGENT']) then that would be decidely not OK, since noagent is one of the most basic blocking criteria.