Magic Browser still with us

[lucy24]

Most recent earlier thread: [webmasterworld.com...]

Abracadabra, it magically converts requests into 403s.

UA: Magic Browser
IP: assorted AWS
Protocol: HTTP

It caught my notice because last month on my test site it made up a whopping 1/4 of all requests. Now, obviously the actual numbers are not huge, it being a test site and all, but I'm not talking about, say, five requests out of a total of 20. Further log-crunching reveals that it first showed its face in April, and has been stopping by every day or so since June.

I mention the protocol because this is an HTTPS site. (I try out everything on the test site first to confirm that the site doesn't explode.) So far, not many malign robots--maybe 1 in 20--come in asking for HTTPS up front. When they do, we will know that it has really become the norm.

[keyplyr]

I see it most every day lately.

So far, not many malign robots--maybe 1 in 20--come in asking for HTTPS up front. When they do, we will know that it has really become the norm.

Maybe... however I wouldn't hold my breath. Most bot runners that use these scraper tools don't care about web standards. They're just after what they can harvest from your site. There are major glitches and nonconformities in most of these tools, but as long as they retrieve the data, why fix them? It's like asking your neighborhood burglar to dress nicer.

[lucy24]

If I didn't need to track redirects, I wouldn't even look at HTTP logs on HTTPS sites. The HTTPS logs are five times as big, and that's where the “real” requests come in.

[keyplyr]

Yup... I watched my HTTP logs for roughly a month after I switched. Now I only take a look if I need to.

My logs get pretty big. I even had to double the ram a couple years ago just so my text editor wouldn't freeze up.