Riddler

[keyplyr]

UA:Riddler (http://riddler.io/about)
Host: Hetzner
5.9.0.0 - 5.9.255.255
5.9.0.0/16
Protocol: HTTP/1.0 for robots.txt, then HTTP/1.1 for HTML
Robots.txt: Yes, then ignored the part where I have it disallowed!

GET /robots.txt HTTP/1.0" 200 1296 "-" "Riddler (http://riddler.io/about)"
GET /example.html HTTP/1.0" 1403 943 "-" "Riddler (http://riddler.io/about)"

[keyplyr]

Touch screen on my phablet reacts oddly to the text editor here at WW. Looks like it moved the "1" from the file size to the response code. Just noticed it. Very strange.

[lucy24]

<tangent>
I see that your 403 response is larger than the robots.txt file (so's mine). Isn't it tempting to just rewrite all of this visitor's requests to serve up nothing but robots.txt, no matter what they ask for? :)

:: uneasily wondering why the 403 responses in my own logs come out to three entirely different sizes, while no two robots.txt are ever exactly the same size (yes, it's always the same physical file) ::
</tangent>

[keyplyr]

wondering why the 403 responses in my own logs come out to three entirely different sizes

Mine do as well... so do everyone's. It is because of packet sizes. They cut-off at different points sometimes. They *should* still add up to the same number of bytes, but often don't.

[tangor]

When I first saw the title of this topic I thought it might be a way to channel Frank Gorshin, the Riddler in the Sixties Batman TV series. Sigh. It's just another 'bot! Drat!

[lucy24]

They *should* still add up to the same number of bytes, but often don't.

What worried me was that there are two size ranges, one twice the size of the other. I had to take a quick look at a random Error Log to make sure I hadn't made a mistake in SSI access control (which would have meant genuinely different content going out on some 403s).

[keyplyr]

I usually have 2 to 3 different sizes, all within 1 byte either direction. One twice the size of the other suggests some type of error.

Are you dynamically creating these 403 pages or just using includes to write content? Maybe the includes are intermittent in efficiency?

You could try to duplicate the discrepancy by using a GET tool and filling in the exact UA and/or referrer. Also notice the HTTP protocol. Maybe it one of these variables that is to blame. If so, then you would know what to fix.

The tool I use is Web-Sniffer from Germany (free.)