homepage Welcome to WebmasterWorld Guest from 184.72.72.182
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
BingBot Fake
Ken_S




msg:4540868
 10:16 am on Jan 31, 2013 (gmt 0)

Looks as though I had me a fake BingBot visit

109.72.87.192 - 109.72.87.255

109.72.87.xxx resolved to hosted.by.pcex-xxx-treme

109.72.87.xxx - - [31/Jan/2013:00:23:56 -0800] "GET / HTTP/1.1" 403 3573 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +ht---tp://www.example.com/bingbot.htm)"

Ken

 

g1smd




msg:4540905
 12:53 pm on Jan 31, 2013 (gmt 0)

In htaccess I block anything that claims to be a popular search engine bot, but is not coming from a small range of specific IPs.

vehMedia




msg:4540912
 1:26 pm on Jan 31, 2013 (gmt 0)

scrapers gunna scrape

Samizdata




msg:4540955
 3:39 pm on Jan 31, 2013 (gmt 0)

scrapers gunna scrape

In this case they appear to have scraped nothing but a 403 page.

Seems like a cunning plan.

...

dstiles




msg:4541118
 10:32 pm on Jan 31, 2013 (gmt 0)

I've seen a lot of bing and google bot UAs from "illegal" IPs over the past few weeks - more than usual but blocks are already in place so not a problem.

lucy24




msg:4541136
 11:42 pm on Jan 31, 2013 (gmt 0)

Careful with the bings. They've been moving into all kinds of obscure new IPs. (Apparently they missed the early-90s /8 handout by about five minutes, so they have to grab IPs where they can find them.) If it's an unfamiliar range and the visitor begins by asking for robots.txt, it may be worth looking them up.

dstiles




msg:4541579
 8:32 pm on Feb 1, 2013 (gmt 0)

The ones I'm seeing are from broadband and known server ranges. They're fake. :)

bigtoga




msg:4541751
 1:45 pm on Feb 2, 2013 (gmt 0)

I bet we had more than 40 IPs just in the past week claiming to be BingBot. Between that and the Majestic12 (MJ12) bot fakes, it's been a fun week.

blend27




msg:4541784
 4:32 pm on Feb 2, 2013 (gmt 0)

Majestic12 are not fakes but the unofficial UFO visitors that haven't gotten the DRILL yet. Like that girl with the shiny teeth on the back of the free local newspaper in your town.

and such....

lucy24




msg:4541864
 11:32 pm on Feb 2, 2013 (gmt 0)

How can you tell if an MJ12 is real or fake? Urk. That sounds like the straight-line to a joke. Last time I looked they didn't have an IP of their own so you had to take them on faith.

wilderness




msg:4541872
 11:45 pm on Feb 2, 2013 (gmt 0)

lucy,
There's old threads by the bot owner who was a forum participant for the the longest while (at least five years).

He set up some type of conformation per Jim's request tailored to your inquiry and/or websites.

In addition, the most widespread of these UA's are the version number he cited as an early version of the software and NOT part of the bots current version.

I've no clue how to locate these old threads.

keyplyr




msg:4541907
 3:59 am on Feb 3, 2013 (gmt 0)

How can you tell if an MJ12 is real or fake?

Isn't MJ12 the one where you sign-up to get SEO data (similar to GWT) in exchange for letting them crawl site? If so, they give you an access code to allow only their authentic bot. I did this for awhile but didn't see the benefit so I stopped allowing them.

lucy24




msg:4541928
 6:25 am on Feb 3, 2013 (gmt 0)

Well, ###. So if I have no idea what you're talking about, it means that every "MJ12" bot I've ever met is fake? :)

:: detour to www site helpfully linked in all UA strings, presumably including the fake ones ::

OK, not so bad. Version numbers seem to fit.

keyplyr




msg:4541938
 7:43 am on Feb 3, 2013 (gmt 0)

Well, ###. So if I have no idea what you're talking about, it means that every "MJ12" bot I've ever met is fake? :)
Not at all.

The access code is just so you can allow the real one and block the others. If I remember correctly, you sign-up for an account at MJ12 and during that process you choose an access code (example: lucy24) then he confirms via email. Then you use that access code in a mod_access filter allowing MJ12 bots that have the code in their UA string, but denying other MJ12 bots that don't. Then I think you need to specifically allow his bot in robots.txt as well (never did understand this part.)

This opt-in filter worked very well. I just discontinued it since I didn't utilize the seo data he offers, you may. He obviously uses your collected site data for business, but at least he is up front about it and offers a trade-off unlike a lot of domain-info-scrapers out there.

blend27




msg:4542006
 1:19 pm on Feb 3, 2013 (gmt 0)

I've no clue how to locate these old threads.


Don,

I think these are the threads on MJ12.

[webmasterworld.com...]
[webmasterworld.com...]

Otherwise,

Most resent from Lord_Majestic:

[webmasterworld.com...]

dstiles




msg:4542131
 10:39 pm on Feb 3, 2013 (gmt 0)

I didn't subscribe to majestic's philosophy and I also decided it was easier to block all MJ bots with both robots.txt (for legit ones) and by 403 (for illegal ones). Have to say the legit ones seem to obey robots.txt.

keyplyr




msg:4542147
 12:12 am on Feb 4, 2013 (gmt 0)



Then you use that access code in a mod_access filter allowing MJ12 bots

Sorry, I meant to say mod_rewrite (not mod_access.)

lucy24




msg:4542161
 1:58 am on Feb 4, 2013 (gmt 0)

... and if you had meant mod_access, you would really have meant mod_authz-thingie ;)

keyplyr




msg:4542173
 3:45 am on Feb 4, 2013 (gmt 0)


It's that thingie that's the real challenge!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved