Gleaning customer interests by crawling referral URLs

Here's a thought for all you B2C site owners...

[dmorison]

Something just occured to me, and I wanted to put it down in bits just to see what people think (it might be rubbish) and also before anyone patents it :)

It's a bit like webbugs, but without the bugs.

Say you operate a very big B2C site; offering products that cover a wide range of interests. Books for example.

If your website becomes anything close to a household name it is very likely that people will link to you without having joined your affiliate program - maybe because they don't know how to - or simply can't be bothered.

So let's say a visitor comes along; and is kind enough to let you know that their referrer was [example.com...]

In theory, you could crawl the referrer and use keywords gleaned from it in order to render a customised homepage for the new visitor.

For example, if you discovered that the referring website mentioned the word "gardening" alot, you could show books about gardening on your home page.

You could do it in one of two ways:

1/ Crawl [example.com...] immediately while serving the HTTP GET operation your new, first time visitors request. You only need to do this the first time you see a new referrer.

~or~

2/ Don't worry about doing immediately and just serve your default homepage. However; add the referring URL to a queue to be crawled later - at which point you could then index the keywords gleaned against their cookie ID so that you can hit them with a more targetted home page next time.

Make sense or complete rubbish?

[jatar_k]

actually it makes complete sense, especially on repeat referals from a single domain. The primary goal being that their visitors are prequalified to be looking for a specific range of products, at least originally.

If you had already spidered the referer then you could serve some customized content, if not add it to the queue so that you can do so next time.

[aaronjf]

I think that is a great idea. Although, would there be time enough to spider the reffering site just before you serve up the page? Seems like it would really slow down the time it would take to serve that visitor up a page.

I am not argueing against that method, I am asking. I have never built a spider and honestly don't know if it could do it fast enough the compile the right content for that visitor.

[amoore]

I have seen similar things being done. Sometimes, when I follow a link from google to a site, the page recognizes that I have come from google and puts something up at the top like "Welcome to example.com all of you people from Google".

I have seen some sites that go so far as to highlight the terms that I searched for in their pages, much like they are highlighted when you view google's cache of their pages. They specialize their pages on the fly to me based on what I was searching for when I got to their site.

I have considered writing a mod_perl module to do that, but just never got around to it. Let me know if you're interested in having me do it.

I have also considered passing the terms searched on to amazon and have them make keyword based advertisements, or whatever they're called. I haven't done that because I have noticed that the keyword based ads that amazon serves to me are never really good. Also, I have a few really technical websites that attract referrals based on terms, abbreviations, and words for which there would be no books from amazon.

I can see specializing your site for referrals more generally, regardless of where they're coming from. You may find that you end up serving some really good pages to people. Please keep us posted on how it turns out.

[dmorison]

Please keep us posted on how it turns out.

Not really anything I can use unfortunately; I run a very boring B2B service!

Might be worth giving B&N a call...!