Fake GoogleBot

[AthlonInside]

I have seen a fake googlebot recently coming to my site. The user agent is googlebot/2.1 but the IPs ain't. A DNS lookup on that IP proof it to be some ISP user.

So what is this guy doing huh?

[marcs]

Trying to detect cloaking based on user-agent only? Not that I use cloaking, but that was the first thing that popped into my mind.

[rfgdxm1]

Checking to see if you are cloaking?

[BigDave]

Sounds like a good user agent for an e-mail harvester. But more likely checking for cloaking.

[Powdork]

Isn't ist simple to spoof user agent with opera?

[GoogleGuy]

Probably not us.. Who knows who it is though--could be anybody faking a user agent.

[AthlonInside]

> Checking to see if you are cloaking?

So I guess it would be a competitor trying to figure out why I rank so high! But he would be dissapointed when he see the same contents! haha.

Actually isn't it a better way to see if a site is cloaking by just viewing the cache in google index?

[anallawalla]

Isn't ist simple to spoof user agent with opera?

How exactly? You can choose between 3-4 browsers but I can't see any place where you can type in an arbitrary user agent.

I am assuming that editing the executable with a hex editor does not work.

[rfgdxm1]

>Actually isn't it a better way to see if a site is cloaking by just viewing the cache in google index?

Not if they block Google from caching.

[rfgdxm1]

>I am assuming that editing the executable with a hex editor does not work.

Actually, it does if you know what you are doing. It isn't hard to spoof user agents if that is what you want to do.

[yetanotheruser]

I've got a funny feeling I saw IE's UA string in the registry somewhere - may be mistaken.. but yeah, about 3 lines of Perl to spoof GB I'd recon.. I'm too naeive to think anyone'd be so lame and dishonest to do it though :( - and I don't know how to spell naeive ;)

[new_shoes]

Yes, hexediting will work, I've done it. If you want to try it, do this:
- make a backup of your browser executable
- edit it with a hexeditor (like ultraedit)
- search for the useragent
- change it, but be sure not to alter the size of the file
works for me.

Something much easier is this:
- get the Mozilla based phoenix browser here [mozilla.org...]
- install it
- get this extension: "User Agent Switcher" [chrispederick.myacen.com]
- the .xpi file is actually a zip file. Extract it. Unpack and alter the .jar file, also as a zip file
- edit the .xul file and add a googlebot menuitem manually
- rezip the package
- open it in Phoenix

will take you a few mins (sticky me for help, if needed), but then you can seemlessly switch to googlebot emulation.

Try to browse around for a little while with your user agent set to googlebot, on sites you know well - it's quite and eye-opener.

[Powdork]

Yeah, Thats what I'd do.;)

<added> whats a hexadecidoohickey?</added>

[new_shoes]

A hexeditor is an editor like notepad, except you can use it to edit executable files.

You can use to change some features in programs, that aren't really meant to be changed - like the name of the program or strings it sends out, but also to circumvent copy protection.

Get a hexeditor here [download.com.com]

Some more info here [i.macmillan.btinternet.co.uk]

[yetanotheruser]

weeee .. haven't done anything like this since my riscOS days :-) Changing UserAgent then was a matter of typing who you wanted to be in the box and clicking ok..

Good though, nice browser that mozilla thingy .. and I got to scare my colleague cos he thought his half done work was getting spidered! ;)

[BigDave]

Many proxies allow you to change your UA.

If you use Konqueror on Linux, you can specify which UA you want to use with different domains. I think that Galeon also allows you to change your UA.

[plasma]

I also recommend a proxy.
I personally use junkbuster. Besides filtering out most banners :) it lets you modify each and every header.
On the other side it lets you protocol the whole traffic making it ideal for testing your cgi, i.e. checking whether a client requests documents based on validators like if-modified-since etc.

[xlcus]

Privoxy also lets you set the user agent.

[AthlonInside]

I have heard of proxy many times but I don't really know what is it. I just recall last time (when using 486) you can add proxy servers to netscape, which are somthing like
proxy.myisp.com. But actually the browser work both with and without this configuration.

So what's a proxy? a firewall?

[anallawalla]

get the Mozilla based phoenix browser here [mozilla.org...]

I stuffed up the UA switcher hack probably through a lack of concentration, but while at the Extensions site, I went to the UAbar author's page (http://uabar.mozdev.org/) and installed UAbar.xpi in Phoenix.

This is a far better solution. It lets you type any arbitrary string as the UA.

In Phoenix you have to go > View > Toolbars > Customize, then drag UAbar where you want it.

[edit]URL added[/edit]