| 7:04 pm on Oct 9, 2003 (gmt 0)|
Many bots deliberately send false UAs & referrers with the intention of getting what a "normal" visitor would see - your plan seems flawed IMHO.
| 7:08 pm on Oct 9, 2003 (gmt 0)|
It may be flawed but in this particular case no bot should ever see this content. Namely, I want to deliver something that would require a human based decision, something that a bot could never do...
Is there no way to get around this?
| 7:18 pm on Oct 9, 2003 (gmt 0)|
|I want to deliver something that would require a human based decision |
I didn't suss this from your initial question - you'll need to add some type of interaction to the page that can't be "botted", like a Turing test - PHP Architect has a really good article on this:
| 8:30 pm on Oct 9, 2003 (gmt 0)|
That's actually a very interesting read.
A turing test isn't exactly what I need however as I still want to deliver some content to the bot. The turing tests I've seen most often on the internet (and described in that article) are based on blocking robots from proceeding. I suppose I could do a time delayed redirect and hope that I've given the human enough time to fill in the turing test while also suggesting to the bot not to index the human content. (Without going into detail I don't want to include bot exclusion for the page).
Doing some reading at WebmasterWorld it also makes sense that search engines really protect against delivering content based upon whether a visitor is a robot/human. I really don't want to get into a non-stop redirect battle and face getting accidentally banned if my attempts are mistaken as malicious cloaking.
Can you suggest any other resources that might be helpful?
Are there any legitimate ways to psuedo wise do:
If you're a robot:
You get content B
elseif you're a human:
You get content A
| 8:59 pm on Oct 9, 2003 (gmt 0)|
I would separate the logic; and do your cloaking the traditional way; based on IP and/or UA.
Simply have your bot test override the referer test; so given that you've got a bot; serve what you want.
Altavista frequently dumps funny URLs in the referer field. I've never seen G'bot present a referer tho'.
serve bot content
if (referer is "")
serve human content A
serve human content B
| 9:52 pm on Oct 9, 2003 (gmt 0)|
|I would separate the logic |
I fully agree, the second portion using http referrers will be quite easy for me. If a bot accidentally slipped through and started spaming random referrers this would be problematic however.
|do your cloaking the traditional way; based on IP and/or UA. |
This is something that's entirely new to me. I've printed off a few threads here, so will get cracking on learning some more.
|Altavista frequently dumps funny URLs in the referer field. |
I took a quick look at my logs but most of the time found scooter was behaving himself/herself. One record I did find a scooter referrer of "http://www.root.mysite.com/category....." which doesn't actually exist. Is this the type of thing you found and have you noticed any patterns?
| 5:19 pm on Oct 12, 2003 (gmt 0)|
| 5:43 pm on Oct 27, 2003 (gmt 0)|
At the same time I'm still curious about the http referrer junk that bots send. I've periodically checked my logs and empirically I've got to agree that at this time googlebots don't send referrers.
Does anyone know if scooter follows any kind of pattern with it's referrer junk dumps? I could then always just add a ereg to just dump any referrers that contained a known junk pattern....