homepage Welcome to WebmasterWorld Guest from 54.227.215.140
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
*missing* - user agent from Yahoo, or spoof?
brokaddr



 
Msg#: 4445812 posted 5:38 am on Apr 26, 2012 (gmt 0)

For a recurring period of time, Yahoo (or what appears to be Yahoo) is sending these peculiar user agents - is this a spoof? They're viewing/indexing everything from pages to images.

One thing that stands out is the hostname, "mrs8.mobile.bf1.yahoo.com" - does Yahoo have some sort of mobile app unrelated to their indexing search engine that allows user to browse pages... I would assume a toolbar could potentially sent this kind of data?

I did try to research the user agent, but for obvious reasons came up empty handed.

Page Viewed: image/example.jpg
Referrer: None
IP: 98.139.242.230
User Agent: *missing*

 

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4445812 posted 8:29 am on Apr 26, 2012 (gmt 0)

Tangential query: Why are they getting through at all? Most people categorically block blank UAs, with maybe a few loopholes such as robots.txt.

Edit: Oh, wait. Do you mean that "missing" is the actual UA string? Lordy, lordy, what will they think of next?

It's definitely a Yahoo IP. But then, they've got more of 'em than you can shake a stick at.

:: shuffling papers ::

8.12.144 Yahoo! Mobile
67.195 Yahoo! Slurp
69.147.64.0/18 yahoo of some sort
72.30 Slurp again
74.6 and more
98.136.0.0/14
-- which I've flagged as including 98.137.72 Slurp and 98.139.241 YahooCacheSystem
184.165 unspecified Yahoo
209.131.32.0/19 Yahoo mobile
209.191.64.0/18 more yahoo

Some of these I've never actually met in person, I just stumbled across them while filling in gaps between humans.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4445812 posted 3:32 pm on Apr 26, 2012 (gmt 0)

I did try to research the user agent, but for obvious reasons came up empty handed.


Pretty much the same response as your inquiry in the Apache forum, unless you provide the a "raw visitor log" line, nobody will be able to assist you.

IMO, the best thing you could is dump the PHP script you enjoy so much, unless your able to modify it to provide useful data and/or identities.

"host name" lookups are cpu server intensive and should be avoided.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4445812 posted 10:00 pm on Apr 26, 2012 (gmt 0)

YahooCacheSystem used to use an actual blank UA, then changed to "YahooCacheSystem" and now has been migrating to "missing"

Seems they read these boards... so soon I expect a new, improved, UA.

brokaddr



 
Msg#: 4445812 posted 2:55 am on Apr 27, 2012 (gmt 0)

Pretty much the same response as your inquiry in the Apache forum, unless you provide the a "raw visitor log" line, nobody will be able to assist you.


The user agent is *missing*. If it were blank, I'd of said it was blank. :)

then changed to "YahooCacheSystem" and now has been migrating to "missing"

So this is a legit user agent for Yahoo? Is there a purpose for them not identifying themselves as yahoo, like they do with the Slurp agent?

And why *missing*? Is there a purpose to this particular bot/user agent?

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4445812 posted 3:05 am on Apr 27, 2012 (gmt 0)


Is there a purpose for them not identifying themselves...

Ya just gotta read this forum. There's lots of info on Yahho Cache System and what they do with the data they mine.

And why *missing*? Is there a purpose to this particular bot/user agent?

As I alluded to above, IMO they are changing the UA to avoid blocks.

brokaddr



 
Msg#: 4445812 posted 3:13 am on Apr 27, 2012 (gmt 0)

Found, thank you: [webmasterworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved