homepage Welcome to WebmasterWorld Guest from 54.205.247.203
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New Googlebot
Nokia6820/2.0
Mokita




msg:403761
 3:02 am on Mar 2, 2006 (gmt 0)

Full UA: Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

coming from 66.249.65.129

I've never seen this in our sites before. I guess it is pretending to be a WAP phone.

I'm wondering why they need a different bot for this? What does it see differently to the normal googlebot?

 

volatilegx




msg:403762
 10:14 pm on Mar 2, 2006 (gmt 0)

Just saw it here, too, coming from 66.249.66.1

Pfui




msg:403763
 8:27 am on Mar 3, 2006 (gmt 0)

I've been wary of that bot and other Google mobiles/WAP phones (Wireless Application Protocol) since early last month when one of its ilk didn't ask for robots.txt and then proceeded to go every which way, hitting my bot traps along the way.

'Twas this UA, from 64.233.166.136, almost like the one you saw:

Nokia3510i/1.0 (05.30) Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)

(Saw that last Nov., too, again from 64.233.166.136; again no robots.txt.)

FWIW...

I'm also not wild about the following info excerpt, emphasis mine, but have yet to specifically opt-out, preferring instead to rewrite G's WAP-related UAs* until the jury's back on whether they routinely ask for, let alone heed robots.txt.

-----
Source: Google Information for Webmasters
Need to remove content from Google's index? [google.com]

[...]

Remove transcoded pages [google.com]

Google Web Search on mobile phones allows users to search all the content in the Google index for desktop web browsers. Because this content isn't written specifically for mobile phones and devices and thus might not display properly, Google automatically translates (or "transcodes") these pages by analyzing the original HTML code and converting it to a mobile-ready format. To ensure that the highest quality and most useable web page is displayed on your mobile phone or device, Google may resize, adjust, or convert images, text formatting and/or certain aspects of web page functionality.

To prevent your web page(s) from being transcoded, please send a removal request to mobile-support@google.com.
-----

Thoughts? About the WAPs? About the transcoding? (The latter makes G's caching look like a walk in the park, copyright-wise.)

.
*Google WAPs

MOT-V171 UP.Browser/6.2.2.7 (GUI) MMP/1.0 UP.Link/6.3.0.0.0 (Google WAP Proxy/1.0)

Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

Nokia3510i/1.0 (05.30) Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)

Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; [google.com...]

thetrasher




msg:403764
 2:11 pm on Mar 3, 2006 (gmt 0)

[webmasterworld.com ]

pfui: "Google WAP Proxy/1.0" is a proxy, not a googlebot.

volatilegx




msg:403765
 5:29 pm on Mar 3, 2006 (gmt 0)

Right. I do not believe this bot is using WAP proxy IP addresses.

Pfui




msg:403766
 7:20 am on Mar 4, 2006 (gmt 0)

Here's my, um, 'rogue robot' philosophy...

"If it looks like a bot, and acts like a bot, and hits bot traps -- it's banned."

: )

Mokita




msg:403767
 2:08 am on Mar 20, 2006 (gmt 0)

I found this information, which confirms it is definitely a bot, similar to Googlebot-Image:

If you go to the newly added robots.txt tab in Google Sitemaps, at the very bottom there is a drop down list. Look inside that, and you'll see this:

Googlebot-Mobile : crawls pages for our mobile index

So I guess it can be banned via robots.txt if required. Presumably (?) it will request and obey it.

Pfui




msg:403768
 5:01 am on Mar 20, 2006 (gmt 0)

From robots.txt:

User-agent: Googlebot-Mobile
Disallow: /

Alas, didn't matter:

crawl-66-249-66-133.googlebot.com
[27/Feb/2006:19:25:25 -0800] "GET /robots.txt
"Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

crawl-66-249-66-133.googlebot.com
[27/Feb/2006:19:25:26 -0800] "GET /
"Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

crawl-66-249-66-133.googlebot.com
[28/Feb/2006:01:47:36 -0800] "GET /welcome.html
"Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

crawl-66-249-66-133.googlebot.com
[28/Feb/2006:01:48:45 -0800] "GET /dir/file.html
"Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

Another bot bites the dust.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved