homepage Welcome to WebmasterWorld Guest from 54.226.21.57
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Mbbf 0.1 Dwl
Wikimedia
keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4478493 posted 6:16 pm on Jul 24, 2012 (gmt 0)



UA: MBBF 0.1 DWL
robots.txt: no


Wikimedia Foundation Inc.
91.198.174.0 - 91.198.174.255
deny from 91.198.174.0/24

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4478493 posted 8:24 pm on Jul 24, 2012 (gmt 0)

I've blocked the whole 91.198.0.0/15 - same "company" but many countries.

First complete blocked was July 2010 with the note:

"Various countries/nets - Wikimedia Foundation Inc and others"

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4478493 posted 10:35 pm on Jul 24, 2012 (gmt 0)

I'm wondering if these hits may also include link checking. I have a dozen or so incoming links from Wikipedia and don't want to loose the traffic.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4478493 posted 9:56 pm on Jul 25, 2012 (gmt 0)

Would you lose traffic by rejecting their bots? As far as I know traffic comes in on its own IP/UA - but I could be wrong. :)

Whatever, I have those IPs blocked and last year one site received a couple of thousand hits from five or six wikis (different countries) (last year was latest stats processed for this site).

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4478493 posted 10:51 pm on Jul 25, 2012 (gmt 0)


Would you lose traffic by rejecting their bots? As far as I know traffic comes in on its own IP/UA


My question was...

I'm wondering if these hits may also include link checking.


Didn't want my incoming links from Wiki* to be invalidated and removed. Once removed it takes a human editor to put it back in. I've had numerous issues with them for years. Every time I find my plagiarized content I remove it, only to see it again a week or two later. Every time I (as an editor) insert my link as a source (since they continue to use my content) the link gets removed by another editor.

It's as bad as the Open Directory. In some areas/topics, it's dominated by only several editors who give outgoing links to favored sites and block any other editors from adding/changing things.

I've pretty much let go of that game but don't want to loose my dozen or so links that remain, thus my concern about this bot being used for link validation.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved