BingBot Fake

[Ken_S]

Looks as though I had me a fake BingBot visit

109.72.87.192 - 109.72.87.255

109.72.87.xxx resolved to hosted.by.pcex-xxx-treme

109.72.87.xxx - - [31/Jan/2013:00:23:56 -0800] "GET / HTTP/1.1" 403 3573 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +ht---tp://www.example.com/bingbot.htm)"

Ken

[g1smd]

In htaccess I block anything that claims to be a popular search engine bot, but is not coming from a small range of specific IPs.

[vehMedia]

scrapers gunna scrape

[Samizdata]

scrapers gunna scrape

In this case they appear to have scraped nothing but a 403 page.

Seems like a cunning plan.

...

[dstiles]

I've seen a lot of bing and google bot UAs from "illegal" IPs over the past few weeks - more than usual but blocks are already in place so not a problem.

[lucy24]

Careful with the bings. They've been moving into all kinds of obscure new IPs. (Apparently they missed the early-90s /8 handout by about five minutes, so they have to grab IPs where they can find them.) If it's an unfamiliar range and the visitor begins by asking for robots.txt, it may be worth looking them up.

[dstiles]

The ones I'm seeing are from broadband and known server ranges. They're fake. :)

[bigtoga]

I bet we had more than 40 IPs just in the past week claiming to be BingBot. Between that and the Majestic12 (MJ12) bot fakes, it's been a fun week.

[blend27]

Majestic12 are not fakes but the unofficial UFO visitors that haven't gotten the DRILL yet. Like that girl with the shiny teeth on the back of the free local newspaper in your town.

and such....

[lucy24]

How can you tell if an MJ12 is real or fake? Urk. That sounds like the straight-line to a joke. Last time I looked they didn't have an IP of their own so you had to take them on faith.

[wilderness]

lucy,
There's old threads by the bot owner who was a forum participant for the the longest while (at least five years).

He set up some type of conformation per Jim's request tailored to your inquiry and/or websites.

In addition, the most widespread of these UA's are the version number he cited as an early version of the software and NOT part of the bots current version.

I've no clue how to locate these old threads.

[keyplyr]

How can you tell if an MJ12 is real or fake?

Isn't MJ12 the one where you sign-up to get SEO data (similar to GWT) in exchange for letting them crawl site? If so, they give you an access code to allow only their authentic bot. I did this for awhile but didn't see the benefit so I stopped allowing them.

[lucy24]

Well, ###. So if I have no idea what you're talking about, it means that every "MJ12" bot I've ever met is fake? :)

:: detour to www site helpfully linked in all UA strings, presumably including the fake ones ::

OK, not so bad. Version numbers seem to fit.

[keyplyr]

Well, ###. So if I have no idea what you're talking about, it means that every "MJ12" bot I've ever met is fake? :)

Not at all.

The access code is just so you can allow the real one and block the others. If I remember correctly, you sign-up for an account at MJ12 and during that process you choose an access code (example: lucy24) then he confirms via email. Then you use that access code in a mod_access filter allowing MJ12 bots that have the code in their UA string, but denying other MJ12 bots that don't. Then I think you need to specifically allow his bot in robots.txt as well (never did understand this part.)

This opt-in filter worked very well. I just discontinued it since I didn't utilize the seo data he offers, you may. He obviously uses your collected site data for business, but at least he is up front about it and offers a trade-off unlike a lot of domain-info-scrapers out there.

[blend27]

I've no clue how to locate these old threads.

Don,

I think these are the threads on MJ12.

[webmasterworld.com...]
[webmasterworld.com...]

Otherwise,

Most resent from Lord_Majestic:

[webmasterworld.com...]

[dstiles]

I didn't subscribe to majestic's philosophy and I also decided it was easier to block all MJ bots with both robots.txt (for legit ones) and by 403 (for illegal ones). Have to say the legit ones seem to obey robots.txt.

[keyplyr]

Then you use that access code in a mod_access filter allowing MJ12 bots

Sorry, I meant to say mod_rewrite (not mod_access.)

[lucy24]

... and if you had meant mod_access, you would really have meant mod_authz-thingie ;)

[keyplyr]

It's that thingie that's the real challenge!