homepage Welcome to WebmasterWorld Guest from 54.205.207.53
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New Googlebot
Nokia6820/2.0
Mokita

5+ Year Member



 
Msg#: 3172 posted 3:02 am on Mar 2, 2006 (gmt 0)

Full UA: Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

coming from 66.249.65.129

I've never seen this in our sites before. I guess it is pretending to be a WAP phone.

I'm wondering why they need a different bot for this? What does it see differently to the normal googlebot?

 

volatilegx

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3172 posted 10:14 pm on Mar 2, 2006 (gmt 0)

Just saw it here, too, coming from 66.249.66.1

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3172 posted 8:27 am on Mar 3, 2006 (gmt 0)

I've been wary of that bot and other Google mobiles/WAP phones (Wireless Application Protocol) since early last month when one of its ilk didn't ask for robots.txt and then proceeded to go every which way, hitting my bot traps along the way.

'Twas this UA, from 64.233.166.136, almost like the one you saw:

Nokia3510i/1.0 (05.30) Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)

(Saw that last Nov., too, again from 64.233.166.136; again no robots.txt.)

FWIW...

I'm also not wild about the following info excerpt, emphasis mine, but have yet to specifically opt-out, preferring instead to rewrite G's WAP-related UAs* until the jury's back on whether they routinely ask for, let alone heed robots.txt.

-----
Source: Google Information for Webmasters
Need to remove content from Google's index? [google.com]

[...]

Remove transcoded pages [google.com]

Google Web Search on mobile phones allows users to search all the content in the Google index for desktop web browsers. Because this content isn't written specifically for mobile phones and devices and thus might not display properly, Google automatically translates (or "transcodes") these pages by analyzing the original HTML code and converting it to a mobile-ready format. To ensure that the highest quality and most useable web page is displayed on your mobile phone or device, Google may resize, adjust, or convert images, text formatting and/or certain aspects of web page functionality.

To prevent your web page(s) from being transcoded, please send a removal request to mobile-support@google.com.
-----

Thoughts? About the WAPs? About the transcoding? (The latter makes G's caching look like a walk in the park, copyright-wise.)

.
*Google WAPs

MOT-V171 UP.Browser/6.2.2.7 (GUI) MMP/1.0 UP.Link/6.3.0.0.0 (Google WAP Proxy/1.0)

Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

Nokia3510i/1.0 (05.30) Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)

Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; [google.com...]

thetrasher

5+ Year Member



 
Msg#: 3172 posted 2:11 pm on Mar 3, 2006 (gmt 0)

[webmasterworld.com ]

pfui: "Google WAP Proxy/1.0" is a proxy, not a googlebot.

volatilegx

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3172 posted 5:29 pm on Mar 3, 2006 (gmt 0)

Right. I do not believe this bot is using WAP proxy IP addresses.

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3172 posted 7:20 am on Mar 4, 2006 (gmt 0)

Here's my, um, 'rogue robot' philosophy...

"If it looks like a bot, and acts like a bot, and hits bot traps -- it's banned."

: )

Mokita

5+ Year Member



 
Msg#: 3172 posted 2:08 am on Mar 20, 2006 (gmt 0)

I found this information, which confirms it is definitely a bot, similar to Googlebot-Image:

If you go to the newly added robots.txt tab in Google Sitemaps, at the very bottom there is a drop down list. Look inside that, and you'll see this:

Googlebot-Mobile : crawls pages for our mobile index

So I guess it can be banned via robots.txt if required. Presumably (?) it will request and obey it.

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3172 posted 5:01 am on Mar 20, 2006 (gmt 0)

From robots.txt:

User-agent: Googlebot-Mobile
Disallow: /

Alas, didn't matter:

crawl-66-249-66-133.googlebot.com
[27/Feb/2006:19:25:25 -0800] "GET /robots.txt
"Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

crawl-66-249-66-133.googlebot.com
[27/Feb/2006:19:25:26 -0800] "GET /
"Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

crawl-66-249-66-133.googlebot.com
[28/Feb/2006:01:47:36 -0800] "GET /welcome.html
"Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

crawl-66-249-66-133.googlebot.com
[28/Feb/2006:01:48:45 -0800] "GET /dir/file.html
"Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

Another bot bites the dust.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved