status code 488 on robots.txt?

[Dan99]

I notice that I get a lot of status code 488s on attempted reads of my robots.txt file. By what I think are mostly halfway reputable bots (such as Twitterbot, commoncrawl), doing a standard GET /robots.txt (that my robots.txt file happens to turn away, along with a number of others). So, um, what's going on?

[lucy24]

Search me. 488? Really? "Not Acceptable Here"-- a response so obscure, I couldn't even find it at w3.org and had to look elsewhere [tools.ietf.org].

Does anyone else find it unnerving that all the technical writeups use telephone terms like "ringing" and "dialing" as if we were all stuck in 1992?

that my robots.txt file happens to turn away, along with a number of others

Are you intentionally turning away requests for robots.txt, or is this collateral damage? Does "others" mean other pages (which you really might not want them to see) or does it mean other robots? I'm trying to figure out whether you're seeing 488 instead of the expected 403, or instead of an expected 200.

:: grasping at straws ::

Have you changed anything about your robots.txt file recently? Or even a non-change, such as re-uploading an unchanged document. Even though 488 falls in the category of "I don't like your face" responses, it can be construed as "robots.txt doesn't like your face, but it's possible that other files on my site are not as particular".

[Dan99]

Yep. it's 488.

199.16.156.126 - - [27/Feb/2015:18:37:26 -0600] "GET /robots.txt HTTP/1.1" 200 488 "-" "Twitterbot/1.0

"

Now, as it turns out, one of my sites has DENY FROM 199.16.156.26 in .htaccess, but that shouldn't prevent Twitterbot from looking at my root robots.txt file.

Also, for example

66.249.75.237 - - [27/Feb/2015:09:17:46 -0600] "GET /robots.txt HTTP/1.1" 200 488 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Now Google is never denied access at all (as far as I can tell).

Whoa. Now that I look at it more carefully, *all* requests for my robots.txt file are now getting 488'ed. Huh?

Could this be an error in my robots.txt file? I don't think I've made any big changes there, and it's a pretty short file. It is owned by Admin, rather than root. Is that right? Would that make a difference?

Now here's something weirder. Looking way back in my logs, a week ago they were also all getting 488s. But 2,3,4,5 weeks ago, they were getting 455s. 6 weeks ago they were getting 425s. I see requests for robots.txt in my logs all the time, but I've never noticed the odd status code on them. Sheesh.

[wilderness]

199.16.156.126 - - [27/Feb/2015:18:37:26 -0600] "GET /robots.txt HTTP/1.1" 200 488

200 is the access code.
488 is the file size

[Dan99]

Oh, now I'm really embarrassed. Thanks.

[wilderness]

Here's an Apache link to the Combine Log Format [httpd.apache.org], however the field are NOT defined in a clear example


This example is clearer and defines the fields separately [publib.boulder.ibm.com]

[Dan99]

No, I know all about the format. My brain just wasn't in gear this morning. My apologies. I am VERY embarrassed.

[lucy24]

:)

It is always good to get a hearty laugh in the course of a weekend. And I've added the world's most comprehensive status-code listing to my bookmarks, so there's that.

[Dan99]

Well, there IS a 488 status code! Like you say, it takes some digging to find it. Now I just have to find a circumstance where I really see it. So there is a (faint) silver lining from this piece of stupidity.

[wilderness]

Dan,
We all have DUH Day and moments.

In my case it simply happens more often ;)

I put the blades on backwards on my tractor/mower and all the grass wouldn't cut right. When I finally realized what I'd done, all I could is laugh at my own DUH.

[tangor]

How cool is it that the robots.txt file size is the same as a status code?

And then provide more fun for webmasters in identifying the more obscure status codes out there?

Yeah, egg on face is one thing, but there's a serendipity silver lining on this one.

[Dan99]

Eh, I just had someone download an mp3 file with a 948562 status code. I dare you to look that one up. Duh!

[lucy24]

I put the blades on backwards on my tractor/mower and all the grass wouldn't cut right.

I really, really wanted this sentence to end "and all the grass ended up longer after I mowed it than before".

[tangor]

Eh, I just had someone download an mp3 file with a 948562 status code. I dare you to look that one up. Duh!

Status Code: 948562 - Foo and Bar mated to create a Snafu, check back later after your kids have paid back their college loans

Results in an endless loop. Requires reset.