Forum Moderators: phranque
The first 301 is due to www canonicalisation of the index page.
*.54.246.68 - - [...] "GET /gone.html HTTP/1.1" 200 298 "http://*seo.*/try.php?u=http://mydomain.*"
You shouldn't be seeing the 301 in your logs.
The 200 status here would appear to be for direct accesses to the error document itself?! ... How is the spammer finding this document?
You shouldn't be seeing the 301 in your logs.
The 200 follows immediately after the 301.
because there shouldn't be a 301 redirect happening at all
Do you see a 410 status anywhere in your logs?
specified a root-relative (starting with a slash) local path to your error document?
I see nothing wrong with the log report. I see different configs at different hosts and they're not all the same.
It is shared hosting, so this issue may be due to some intentional or unintentional configuration of the server
may I send you the RewriteCond %{HTTP_REFERER} section of my .htaccess by Stickymail?
The whole file is too big.
RewriteRule !^gone\.html - [G] . No slash. The error document is in the root.
ErrorDocument 410 /gone.html
ErrorDocument 410 http://example.com/gone.html
Let's not post sensitive server info in the public forum. Best to do that privately with someone you trust.may I send you the RewriteCond %{HTTP_REFERER} section of my .htaccess by Stickymail?You can, but why not post it in the forum?
Options +FollowSymLinks
RewriteEngine on
ErrorDocument 404 /notfound.html
ErrorDocument 410 /gone.html
RewriteCond %{REMOTE_ADDR} ^*\.(9[2-5])\. [OR]
...
RewriteRule !^gone\.html - [G]
RewriteCond %{HTTP_REFERER} seo [NC,OR]
...
RewriteRule !^gone\.html - [G]
RewriteCond %{HTTP_USER_AGENT} * [NC,OR]
...
RewriteRule !^gone\.html - [G]
RewriteRule *|* - [L]
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?myowndomain\.*(/)?.*$ [NC]
RewriteCond %{HTTP_REFERER} !^https://(www\.)?google\. #plus a few others and variations#
...
RewriteRule \.(jpeg?|jpg|gif|wav|pdf|rtf)$ http://www.myowndomain.*/*.png [NC,R,L]
RewriteCond %{REQUEST_URI} ^([^.]+\.html)
RewriteRule \.html. http://www.myowndomain.*/$1 [R=301,L]
RewriteCond %{HTTP_HOST} ^((www\.)?(*|*|*)|*)\.* [NC]
RewriteRule (.*) http://www.myowndomain.*/$1 [R=301,L]
Many. But they are all due to blocking by IP or by UA. It is blocking by referrer that is not working as it should.
Here is the abridged and depersonalised (with *) .htaccess file.
RewriteRule (.*) http://www.myowndomain.*/$1 [R=301,L]
Please do not post specific details such as domain names, full IP addresses, or personally-identifiable information such as name, e-mail address, IM screen name, etc. Such specifics will be edited or removed in accordance with our Terms of Service [webmasterworld.com...] which may render your post meaningless. Please replace all instances of your domain name with "example.com" before posting.
/try.php?u=
Revealing my domain would be considered self-promotion. Revealing banned IP:s or referrers could offend someone.
RewriteCond %{REMOTE_ADDR} ^*\.(9[2-5])\. [OR]
# 1
RewriteCond %{REMOTE_ADDR} ^*\.(9[2-5])\. [OR]
...
RewriteRule !^gone\.html - [G]
# 2
RewriteCond %{HTTP_REFERER} seo [NC,OR]
...
RewriteRule !^gone\.html - [G]
# 3
RewriteCond %{HTTP_USER_AGENT} * [NC,OR]
...
RewriteRule !^gone\.html - [G]
MSG4801456: <action type="AbortRequest" /> that is on IIS.
That is, however, not always so. The interval is sometimes longer. .... intervals of up to 8 seconds. That is not how mod_rewrite works.
whitespace said: "The 200 status here would appear to be for direct accesses to the error document itself?!"
As the rule is !^gone\.html - [G] a direct request for /gone.html will inevitably result in a 200.
# Exception "built-in" to the one directive
RewriteRule !^gone\.html - [G]
# Exception for the error document
RewriteRule ^gone\.html - [G]
# ...lots of other stuff here...
RewriteRule .* - [G]
RewriteRule *|* - [L]
I have for the time being reverted to .* - [G]. Let us see if this eliminates the spammer's 301s.
But the other two rulesets for IP and UA blocking (on either side of the Referer block) use !^gone\.html - [G] and they appear to work OK?
*.242.246.23 - - [27/Aug/2016:13:59:56] "GET / HTTP/1.1" 500 - "http://pizza-tycoon.*/" "Mozilla*"
No 500 loop
Request exceeded the limit of 10 internal redirects due to probable configuration error.
No 301 hit.
the user (although a bot in this case) may well see, what looks like, a system generated 410 error page - but this will be served with a 500 status with an additional note along the lines of: "Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request."
the user (although a bot in this case) may well see, what looks like, a system generated 410 error page - but this will be served with a 500 status with an additional note along the lines of: "Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request."
Why cant You try
RewriteCond %{HTTP_REFERER} (seo|pizza) [NC]
RewriteRule .? /errorfolder/410.php [L]
Yes, that is what happens when the rule is simply .* - [G]. I have seen it all on the screen myself when testing. But why? If Apache has generated its standard 410 page, why does it additionally look for a custom 410 ErrorDocument? If there is none a 500 will result.
But I do have a custom 410 page. Why does Apache not find it?
It looks like the host in mid May made some server reconfigurations causing this new problem.
ErrorDocument default
if you don't have a custom ErrorDocument defined and are instead just relying on the default built-in Apache error message, then you don't get a rewrite loop (and no 500 error). You simply get the default 410 page returned with a 410 status and 410 is recorded in the access log.
if the host declared a custom ErrorDocument in the server config then a lone RewriteRule .* - [G] directive in your .htaccess file could potentially start generating these rewrite loops and 500 errors recorded in your access log.
In my previous thread I presented the only explanation I have gotten from the host (verbatim translation):
"RewriteRule .* - [G] makes the server look also within the .htaccess area for a custom 410 error document [which I did not have at that time], in doing which another 410 is caused as well as a loop resulting in a 500 error in the log".
They suggested I use the following rule:
RewriteRule foobar.html - [G]
which will produce a 404 in the log, they said.
If I next define
RewriteRule foobar.html - [G]
ErrorDocument 410 /410.html
a normal 410 will be logged, they said.
I am unable to understand why that works fine for blocking by IP and by UA but not for blocking by referrer.
...replaced its declaration with ErrorDocument default ... So far everything has worked fine! No 500s, and of course no code 200 hits on an error document.
Who can explain this?
