homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

lynx robot
too much information?

 4:14 am on Jul 15, 2012 (gmt 0)

Query: If I didn't have external evidence*, would it be obvious that this is a robot?

Lynx/2.8.5rel.1 libwww-FM/2.14FC SSL-MM/1.4.1b OpenSSL/0.9.7d-dev

For comparison purposes I visited the same page myself in Lynx and checked the log. It says only

Lynx/2.8.6rel.5 libwww-FM/2.14

Is this one of those stereotypical police scenarios where you sit back and let them hang themselves? It should have stopped at 2.14FC :) (Interesting that real Lynx doesn't send any information about computer or OS. Never knew that.)

* It visited the same page twice, almost concurrently-- within the same second in logs-- using two different referers. I will concede, Don, that this is as close to physically impossible as you can get. And then proceeded to the home page in less time than it would have taken a human to find or type the link.



 5:18 am on Jul 15, 2012 (gmt 0)

Just because it might be the real Lynx doesn't mean it's not being driven to read pages by a bot script, pretty clever really.

[edited by: incrediBILL at 5:55 am (utc) on Jul 15, 2012]


 5:53 am on Jul 15, 2012 (gmt 0)

There's an online tool that generates that UA.
Anybody may use.

Lynx Viewer [delorie.com]

The intent is to see the underlying-HTML in the same manner that bots do.

I use to use the thing on my own sites frequently some years back, however had the UA denied when I wasn't using it.


 5:59 am on Jul 15, 2012 (gmt 0)

Ha, that tool doesn't work for my sites.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved