Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Bot, Robot, Spider, Crawler from Brazil lucy24
This was so adorable, I had to share.
184.108.40.206 - - [25/Apr/2014:07:49:30 -0700] "GET /robots.txt HTTP/1.1" 200 779 "-" "WIRE/0.22 (Linux; x86_64; Bot,Robot,Spider,Crawler)"
220.127.116.11 - - [25/Apr/2014:12:48:13 -0700] "GET /robots.txt HTTP/1.1" 200 779 "-" "WIRE/0.22 (Linux; x86_64; Bot,Robot,Spider,Crawler)" ... and that was all she wrote. D'you think it might possibly be a robot?
Covering all the bases. webcentric
It's a human trying to make you think it's a robot. Don't be fooled. Write an algorithm to let her in. If I'm right, she is sleepwalking and will walk straight into a wall. Then you can sign her up for your newsletter and sell her a bottle of aspirin. brotherhood of LAN
Perhaps a test to discover that 99.9% of sites would happily let that continue crawling. webcentric
Perhaps a test to discover that 99.9% of sites would happily let that continue crawling. I was thinking of the exceptions I'd have to configure to be able to let that in for a crawl. ;) lucy24
Hey, I just met its brother (URL lightly obfuscated because it's Brazil and might be an infected human machine):
177.32.69.abc - - [09/Jun/2014:17:26:02 -0700] "GET /robots.txt HTTP/1.1" 200 802 "-" "WIRE/0.22 (Linux; x86_64; Bot-Robot-Spider-Crawler)"
Y'know, sometimes it's just so hard to figure out if something is a robot or not...