homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Genieo Web Filter

 6:36 pm on Jun 19, 2012 (gmt 0)

UA: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html
robots.txt: no

Scrapes Twitter for links.
Comes from various IP addresses.
UA can also display during visit at varied times.

Seems be a security or parental filter plugged into the user's browser as well as a stand alone crawler building a database.

I've been blocking it for 3 weeks. Anyone have additional info?



 3:46 am on Jun 20, 2012 (gmt 0)

Notice the missing closing parenthesis.


 5:59 am on Jun 20, 2012 (gmt 0)

Logs say:

Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html

(with, yup, an odd number of parentheses)

Site says:

Mozilla/5.0 (compatible; Genieo/x.x http://www.genieo.com/webfilter.html)

We at Genieo, [**sic comma] design our client software to access only pages which are reasonably likely to interest the user and appear on their homepage.

followed by a bunch of blahblah which interests me much less than the question of how they prevented my browser from showing a horizontal scroll bar even when their text didn't fit. But I guess that's for a different forum.

Picked up my home page, which nobody ever goes to, so no idea what they're up to.

whois tells me only that the most recent IP is from a /12 range belonging to qwest-- which I already knew, because there are humans in there.


 7:56 am on Jun 20, 2012 (gmt 0)

Hits me about 30 times a day, always just HTML, always 403. Pretty stupid.


 11:04 am on Oct 3, 2012 (gmt 0)


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved