Forum Moderators: open

Message Too Old, No Replies

Yahoo Nokia Symbian crawler

Is this a Yahoo crawler for mobile aps?

         

grandma genie

11:43 pm on Oct 8, 2010 (gmt 0)

10+ Year Member



Just a question. Is this Yahoo? Came on the site at 4 a m and left at 2 p m. Are they crawling for phone users? Does this go to the Yahoo search engine? Wasn't sure if it was the Yahoo crawler or someone using Yahoo to do a crawl anonymously. Are these types of crawls generally welcome by websites?

209.191.123.nn - - [08/Oct/2010:04:10:53 -0400] "GET /lizards.html HTTP/1.0" 304 - "-" "Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; h**p://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)"

Pfui

12:13 am on Oct 9, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



And now, from the Sweat Equity/DIY Department:) --

1.) Did you WHOIS the IP address? I show huge chunks (if not all) of 209.191. to be Yahoo.

2.) See the info link in the UA string for answers: [help.yahoo.com...]

FWIW: All of the hosts I've seen from similar .search.mud.yahoo.com Hosts* using the exact same UA requested/respected robots.txt. Yours, too?

*For example: 209.191.82.245 = mssn01.search.mud.yahoo.com

grandma genie

1:33 am on Oct 9, 2010 (gmt 0)

10+ Year Member



I just typed the whole string into Google and found a site that lists all Yahoo's user agents, and this one is there. It says: These user agent strings belong to Yahoo!’s mobile web index crawler. I don't have a mobile site and don't know what my site would look like on a mobile device. So, the bottom line is, if they want to crawl, that is fine. I guess they have been around for awhile, but I am just now seeing them crawl my site. I usually see Yahoo Slurp. By the way, the reverse DNS lookup is cr04.wap.search.mud.yahoo.com. Thanks Pfui.