homepage Welcome to WebmasterWorld Guest from 54.205.241.107
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Bot, Robot, Spider, Crawler from Brazil
lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4666339 posted 10:25 pm on Apr 27, 2014 (gmt 0)

This was so adorable, I had to share.

201.81.177.130 - - [25/Apr/2014:07:49:30 -0700] "GET /robots.txt HTTP/1.1" 200 779 "-" "WIRE/0.22 (Linux; x86_64; Bot,Robot,Spider,Crawler)"
201.81.177.130 - - [25/Apr/2014:12:48:13 -0700] "GET /robots.txt HTTP/1.1" 200 779 "-" "WIRE/0.22 (Linux; x86_64; Bot,Robot,Spider,Crawler)"


... and that was all she wrote. D'you think it might possibly be a robot?

 

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4666339 posted 3:45 am on Apr 28, 2014 (gmt 0)

Covering all the bases.

webcentric

WebmasterWorld Senior Member Top Contributors Of The Month



 
Msg#: 4666339 posted 3:59 am on Apr 28, 2014 (gmt 0)

It's a human trying to make you think it's a robot. Don't be fooled. Write an algorithm to let her in. If I'm right, she is sleepwalking and will walk straight into a wall. Then you can sign her up for your newsletter and sell her a bottle of aspirin.

brotherhood of LAN

WebmasterWorld Administrator brotherhood_of_lan us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4666339 posted 4:00 am on Apr 28, 2014 (gmt 0)

Perhaps a test to discover that 99.9% of sites would happily let that continue crawling.

webcentric

WebmasterWorld Senior Member Top Contributors Of The Month



 
Msg#: 4666339 posted 2:36 pm on Apr 28, 2014 (gmt 0)

Perhaps a test to discover that 99.9% of sites would happily let that continue crawling.


I was thinking of the exceptions I'd have to configure to be able to let that in for a crawl. ;)

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4666339 posted 7:31 pm on Jun 10, 2014 (gmt 0)

Hey, I just met its brother (URL lightly obfuscated because it's Brazil and might be an infected human machine):

177.32.69.abc - - [09/Jun/2014:17:26:02 -0700] "GET /robots.txt HTTP/1.1" 200 802 "-" "WIRE/0.22 (Linux; x86_64; Bot-Robot-Spider-Crawler)"

Y'know, sometimes it's just so hard to figure out if something is a robot or not...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved