homepage Welcome to WebmasterWorld Guest from 54.242.241.20
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Genieo Web Filter
keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4467311 posted 6:36 pm on Jun 19, 2012 (gmt 0)


UA: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html
robots.txt: no

Scrapes Twitter for links.
Comes from various IP addresses.
UA can also display during visit at varied times.

Seems be a security or parental filter plugged into the user's browser as well as a stand alone crawler building a database.

I've been blocking it for 3 weeks. Anyone have additional info?

 

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4467311 posted 3:46 am on Jun 20, 2012 (gmt 0)


Notice the missing closing parenthesis.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4467311 posted 5:59 am on Jun 20, 2012 (gmt 0)

Logs say:

Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html

(with, yup, an odd number of parentheses)

Site says:

Mozilla/5.0 (compatible; Genieo/x.x http://www.genieo.com/webfilter.html)

and
We at Genieo, [**sic comma] design our client software to access only pages which are reasonably likely to interest the user and appear on their homepage.


followed by a bunch of blahblah which interests me much less than the question of how they prevented my browser from showing a horizontal scroll bar even when their text didn't fit. But I guess that's for a different forum.

Picked up my home page, which nobody ever goes to, so no idea what they're up to.

whois tells me only that the most recent IP is from a /12 range belonging to qwest-- which I already knew, because there are humans in there.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4467311 posted 7:56 am on Jun 20, 2012 (gmt 0)


Hits me about 30 times a day, always just HTML, always 403. Pretty stupid.

fips



 
Msg#: 4467311 posted 11:04 am on Oct 3, 2012 (gmt 0)

[mugo.ca...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved