homepage Welcome to WebmasterWorld Guest from 54.166.84.82
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
BingBot Fake
Ken_S



 
Msg#: 4540866 posted 10:16 am on Jan 31, 2013 (gmt 0)

Looks as though I had me a fake BingBot visit

109.72.87.192 - 109.72.87.255

109.72.87.xxx resolved to hosted.by.pcex-xxx-treme

109.72.87.xxx - - [31/Jan/2013:00:23:56 -0800] "GET / HTTP/1.1" 403 3573 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +ht---tp://www.example.com/bingbot.htm)"

Ken

 

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4540866 posted 12:53 pm on Jan 31, 2013 (gmt 0)

In htaccess I block anything that claims to be a popular search engine bot, but is not coming from a small range of specific IPs.

vehMedia



 
Msg#: 4540866 posted 1:26 pm on Jan 31, 2013 (gmt 0)

scrapers gunna scrape

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4540866 posted 3:39 pm on Jan 31, 2013 (gmt 0)

scrapers gunna scrape

In this case they appear to have scraped nothing but a 403 page.

Seems like a cunning plan.

...

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4540866 posted 10:32 pm on Jan 31, 2013 (gmt 0)

I've seen a lot of bing and google bot UAs from "illegal" IPs over the past few weeks - more than usual but blocks are already in place so not a problem.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4540866 posted 11:42 pm on Jan 31, 2013 (gmt 0)

Careful with the bings. They've been moving into all kinds of obscure new IPs. (Apparently they missed the early-90s /8 handout by about five minutes, so they have to grab IPs where they can find them.) If it's an unfamiliar range and the visitor begins by asking for robots.txt, it may be worth looking them up.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4540866 posted 8:32 pm on Feb 1, 2013 (gmt 0)

The ones I'm seeing are from broadband and known server ranges. They're fake. :)

bigtoga

5+ Year Member



 
Msg#: 4540866 posted 1:45 pm on Feb 2, 2013 (gmt 0)

I bet we had more than 40 IPs just in the past week claiming to be BingBot. Between that and the Majestic12 (MJ12) bot fakes, it's been a fun week.

blend27

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4540866 posted 4:32 pm on Feb 2, 2013 (gmt 0)

Majestic12 are not fakes but the unofficial UFO visitors that haven't gotten the DRILL yet. Like that girl with the shiny teeth on the back of the free local newspaper in your town.

and such....

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4540866 posted 11:32 pm on Feb 2, 2013 (gmt 0)

How can you tell if an MJ12 is real or fake? Urk. That sounds like the straight-line to a joke. Last time I looked they didn't have an IP of their own so you had to take them on faith.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4540866 posted 11:45 pm on Feb 2, 2013 (gmt 0)

lucy,
There's old threads by the bot owner who was a forum participant for the the longest while (at least five years).

He set up some type of conformation per Jim's request tailored to your inquiry and/or websites.

In addition, the most widespread of these UA's are the version number he cited as an early version of the software and NOT part of the bots current version.

I've no clue how to locate these old threads.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4540866 posted 3:59 am on Feb 3, 2013 (gmt 0)

How can you tell if an MJ12 is real or fake?

Isn't MJ12 the one where you sign-up to get SEO data (similar to GWT) in exchange for letting them crawl site? If so, they give you an access code to allow only their authentic bot. I did this for awhile but didn't see the benefit so I stopped allowing them.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4540866 posted 6:25 am on Feb 3, 2013 (gmt 0)

Well, ###. So if I have no idea what you're talking about, it means that every "MJ12" bot I've ever met is fake? :)

:: detour to www site helpfully linked in all UA strings, presumably including the fake ones ::

OK, not so bad. Version numbers seem to fit.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4540866 posted 7:43 am on Feb 3, 2013 (gmt 0)

Well, ###. So if I have no idea what you're talking about, it means that every "MJ12" bot I've ever met is fake? :)
Not at all.

The access code is just so you can allow the real one and block the others. If I remember correctly, you sign-up for an account at MJ12 and during that process you choose an access code (example: lucy24) then he confirms via email. Then you use that access code in a mod_access filter allowing MJ12 bots that have the code in their UA string, but denying other MJ12 bots that don't. Then I think you need to specifically allow his bot in robots.txt as well (never did understand this part.)

This opt-in filter worked very well. I just discontinued it since I didn't utilize the seo data he offers, you may. He obviously uses your collected site data for business, but at least he is up front about it and offers a trade-off unlike a lot of domain-info-scrapers out there.

blend27

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4540866 posted 1:19 pm on Feb 3, 2013 (gmt 0)

I've no clue how to locate these old threads.


Don,

I think these are the threads on MJ12.

[webmasterworld.com...]
[webmasterworld.com...]

Otherwise,

Most resent from Lord_Majestic:

[webmasterworld.com...]

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4540866 posted 10:39 pm on Feb 3, 2013 (gmt 0)

I didn't subscribe to majestic's philosophy and I also decided it was easier to block all MJ bots with both robots.txt (for legit ones) and by 403 (for illegal ones). Have to say the legit ones seem to obey robots.txt.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4540866 posted 12:12 am on Feb 4, 2013 (gmt 0)



Then you use that access code in a mod_access filter allowing MJ12 bots

Sorry, I meant to say mod_rewrite (not mod_access.)

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4540866 posted 1:58 am on Feb 4, 2013 (gmt 0)

... and if you had meant mod_access, you would really have meant mod_authz-thingie ;)

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4540866 posted 3:45 am on Feb 4, 2013 (gmt 0)


It's that thingie that's the real challenge!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved